[opensuse] Huge memory ussage by spamd.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4034 root 20 0 1295260 1,146g 2780 S 0,000 14,66 8:07.59 spamd child 4035 root 20 0 1206876 1,061g 2908 S 0,000 13,58 4:09.77 spamd child 1 gigabyte resident size? ps afxu: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 27884 0.0 0.8 153440 66892 ? Ss May04 0:06 /usr/sbin/spamd -d -c --max-children=8 -r /var/run/spamd.pid root 4034 1.1 14.6 1295260 1201160 ? S May04 8:07 \_ spamd child root 4035 0.5 13.5 1206876 1112952 ? S May04 4:09 \_ spamd child Current date is Mon May 5 11:42:23 CEST 2014, so it has been running just about 11 hours. After restarting it: top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11614 root 20 0 153360 68300 2996 S 0,000 0,834 0:01.31 /usr/sbin/spamd 11617 root 20 0 153360 65860 556 S 0,000 0,804 0:00.00 spamd child 11618 root 20 0 153360 65844 540 S 0,000 0,804 0:00.00 spamd child ps afxu: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 11614 6.9 0.8 153360 68300 ? Ss 11:35 0:01 /usr/sbin/spamd -d -c --max-children=8 -r /var/run/spamd.pid root 11617 0.0 0.8 153360 65860 ? S 11:35 0:00 \_ spamd child root 11618 0.0 0.8 153360 65844 ? S 11:35 0:00 \_ spamd child At about this time, I see this in the log: <2.4> 2014-05-05 11:29:37 Telcontar spamd 4034 - - message repeated 3 times: [ Use of each() on hash after insertion without resetting hash iterator results in undefined behavior, Perl interpreter: 0x23ce010 at /usr/lib/perl5/vendor_perl/5.18.1/Mail/SpamAssassin/AsyncLoop.pm line 363.] <2.6> 2014-05-05 11:32:31 Telcontar spamd 4034 - - spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 bytes. <2.6> 2014-05-05 11:32:31 Telcontar spamd 4034 - - spamd: result: Y 6 - BAYES_50,DKIM_ADSP_CUSTOM_MED,DRUGS_MUSCLE,FREEMAIL_FROM,FUZZY_XPILL,HTML_MESSAGE,LOTS_OF_MONEY,NML_ADSP_CUSTOM_MED,NORMAL_HTTP_TO_IP,NO_RELAYS,OBFUSCATING_COMMENT,T_FRT_COCK,T_FRT_FUCK1,WEIRD_QUOTING scantime=213.1,size=15976538,user=cer,uid=1000,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=34299,mid=<CA+jbDYSGA7ws6hGgo_Ehx+czKipYjntzqbr7ZFSw6BZO+Rkw6A@mail.gmail.com>,bayes=0.499720,autolearn=disabled <2.6> 2014-05-05 11:32:31 Telcontar postfix 11205 - - 7CCCE61821: to=<cer@localhost.valinor>, relay=local, delay=221, delays=7.9/0/0/214, dsn=2.0.0, status=sent (delivered to command: /usr/bin/procmail) <2.6> 2014-05-05 11:32:31 Telcontar postfix 28651 - - 7CCCE61821: removed <2.6> 2014-05-05 11:32:31 Telcontar spamd 27884 - - message repeated 2 times: [ prefork: child states: II] ... <2.6> 2014-05-05 11:35:45 Telcontar spamd 27884 - - spamd: server killed by SIGTERM, shutting down I may undertand the size increase while processing a problematic email but the size should reduce itself after finishing with it. It did not. Ah, and that email is not spam. It is a dump of a WhatsApp chat with multimedia attachments. - -- Cheers Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlNnYGgACgkQtTMYHG2NR9UmggCfWTWk+4iQm2uvcd5Ewu5M62Sy juwAn0l69J88Sn+Hfb0Avd1hD8+KiVB8 =/HXT -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Carlos E. R. <carlos.e.r@opensuse.org> [05-05-14 05:57]:
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4034 root 20 0 1295260 1,146g 2780 S 0,000 14,66 8:07.59 spamd child 4035 root 20 0 1206876 1,061g 2908 S 0,000 13,58 4:09.77 spamd child
1 gigabyte resident size?
ps afxu:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 27884 0.0 0.8 153440 66892 ? Ss May04 0:06 /usr/sbin/spamd -d -c --max-children=8 -r /var/run/spamd.pid root 4034 1.1 14.6 1295260 1201160 ? S May04 8:07 \_ spamd child root 4035 0.5 13.5 1206876 1112952 ? S May04 4:09 \_ spamd child
Current date is Mon May 5 11:42:23 CEST 2014, so it has been running just about 11 hours.
After restarting it:
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11614 root 20 0 153360 68300 2996 S 0,000 0,834 0:01.31 /usr/sbin/spamd 11617 root 20 0 153360 65860 556 S 0,000 0,804 0:00.00 spamd child 11618 root 20 0 153360 65844 540 S 0,000 0,804 0:00.00 spamd child
ps afxu:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 11614 6.9 0.8 153360 68300 ? Ss 11:35 0:01 /usr/sbin/spamd -d -c --max-children=8 -r /var/run/spamd.pid root 11617 0.0 0.8 153360 65860 ? S 11:35 0:00 \_ spamd child root 11618 0.0 0.8 153360 65844 ? S 11:35 0:00 \_ spamd child
??, but on my box I see: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 4968 0.0 0.1 135596 61880 ? Ss May04 0:05 /usr/sbin/spamd -d -c -L -r /var/run/spamd.pid root 5200 0.0 0.1 135596 60204 ? S May04 0:00 \_ spamd child root 5202 0.0 0.1 135596 60204 ? S May04 0:00 \_ spamd child -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://linuxcounter.net -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4034 root 20 0 1295260 1,146g 2780 S 0,000 14,66 8:07.59 spamd child 4035 root 20 0 1206876 1,061g 2908 S 0,000 13,58 4:09.77 spamd child
1 gigabyte resident size?
ps afxu:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 27884 0.0 0.8 153440 66892 ? Ss May04 0:06 /usr/sbin/spamd -d -c --max-children=8 -r /var/run/spamd.pid root 4034 1.1 14.6 1295260 1201160 ? S May04 8:07 \_ spamd child root 4035 0.5 13.5 1206876 1112952 ? S May04 4:09 \_ spamd child
Current date is Mon May 5 11:42:23 CEST 2014, so it has been running just about 11 hours.
Here I show: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 2363 0.0 0.0 167108 66548 ? Ss Apr28 2:20 /usr/bin/spamd --syslo root 99955 0.0 0.0 167108 63612 ? S Apr29 0:00 \_ spamd child root 99882 0.0 0.0 167108 63604 ? S Apr30 0:00 \_ spamd child root 99960 0.0 0.0 167108 63592 ? S May01 0:00 \_ spamd child root 99917 0.0 0.0 167108 63592 ? S 08:45 0:00 \_ spamd child root 95199 0.0 0.0 167108 63592 ? S 12:10 0:00 \_ spamd child root 96710 0.0 0.0 167108 63676 ? S 12:14 0:00 \_ spamd child root 98404 0.0 0.0 167108 63628 ? S 12:17 0:00 \_ spamd child ---- That's after nearly a week of uptime & 2h20m of cpu time, but note, my command line has: root 2363 0.0 0.0 167108 66548 ? Ss Apr28 2:20 /usr/bin/spamd --syslog=local7 --allow-tell --create-prefs --daemonize --max-children=24 --min-spare=6 --max-conn-per-child=1 --user-config --listen-ip=127.0.0.1 --allowed-ips=192.168.3.0/24,127.0.0.1/32,192.168.4.0/24 --ipv4 -g spamd -r /var/run/spamd.pid ** Significantly: "--max-conn-per-child=1"
I may undertand the size increase while processing a problematic email but the size should reduce itself after finishing with it. It did not.
Where is 'this' (size reduction) documented? That may be the prob.... -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday, 2014-05-05 at 12:29 -0700, Linda Walsh wrote:
Carlos E. R. wrote:
1 gigabyte resident size?
ps afxu:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 27884 0.0 0.8 153440 66892 ? Ss May04 0:06 /usr/sbin/spamd -d -c --max-children=8 -r /var/run/spamd.pid root 4034 1.1 14.6 1295260 1201160 ? S May04 8:07 \_ spamd > child root 4035 0.5 13.5 1206876 1112952 ? S May04 4:09 \_ spamd child
Current date is Mon May 5 11:42:23 CEST 2014, so it has been running just about 11 hours.
That's after nearly a week of uptime & 2h20m of cpu time, but note, my command line has:
root 2363 0.0 0.0 167108 66548 ? Ss Apr28 2:20 /usr/bin/spamd --syslog=local7 --allow-tell --create-prefs --daemonize --max-children=24 --min-spare=6 --max-conn-per-child=1 --user-config --listen-ip=127.0.0.1 --allowed-ips=192.168.3.0/24,127.0.0.1/32,192.168.4.0/24 --ipv4 -g spamd -r /var/run/spamd.pid
** Significantly: "--max-conn-per-child=1"
It is not in the spamd manual. There is this paragraph in the man: See the README file in the "spamd" directory of the SpamAssassin distribution for more details. I see "/usr/share/doc/packages/spamassassin/README", but it also says nothing of that variable.
I may undertand the size increase while processing a problematic email but the size should reduce itself after finishing with it. It did not.
Where is 'this' (size reduction) documented? That may be the prob....
Dunno, it stands to reason. Ok, see here: /usr/share/doc/packages/spamassassin/README: When spamd receives a connection, it spawns a child to handle the request. The child will expect to read an email message from the network socket, which should then be closed for writing on the other end (so spamd receives an EOF). spamd will then use SA to rewrite the message, and dump the processed message back to the socket before closing the connection. The child process then dies. But that paragraph has this prefix, which makes it obsolete: /* * FIXME: The following paragraph(s) have to be updated as childs are now * pre-spawned */ So the entire documentation is wrong. - -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlNpXAcACgkQtTMYHG2NR9WJLACfaYAX+4Id1/avR7964jDXX5IW 1HsAn3RwUYtU0n99vfltHOgYXC+wmj4Q =lAyy -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On Monday, 2014-05-05 at 12:29 -0700, Linda Walsh wrote:
** Significantly: "--max-conn-per-child=1"
It is not in the spamd manual.
It is in mine, but my spamassassin is backlevel and built from source: NAME spamd - daemonized version of spamassassin SYNOPSIS spamd [options] Options: [snip] -m num, --max-children=num Allow maximum num children --min-children=num Allow minimum num children --min-spare=num Lower limit for number of spare children --max-spare=num Upper limit for number of spare children --max-conn-per-child=num Maximum connections accepted by child before it is respawned -- Per Jessen, Zürich (12.6°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/07/2014 12:37 AM, Per Jessen wrote:
Carlos E. R. wrote:
On Monday, 2014-05-05 at 12:29 -0700, Linda Walsh wrote:
** Significantly: "--max-conn-per-child=1"
It is not in the spamd manual.
It is in mine, but my spamassassin is backlevel and built from source:
NAME spamd - daemonized version of spamassassin
SYNOPSIS spamd [options]
Options:
[snip] -m num, --max-children=num Allow maximum num children --min-children=num Allow minimum num children --min-spare=num Lower limit for number of spare children --max-spare=num Upper limit for number of spare children --max-conn-per-child=num Maximum connections accepted by child before it is respawned
So what's the significance of this? Like many others here I run a single user system, its just me who who uses it. I bring in my mail using fetchmail which feeds into procmail. The procmail makes use of spamc, the client that calls on spamd. It processes one mail message at a time. I can see that by watching the logs with 'tail -f'. I emphasise: ONE MESSAGE AT A TIME. There is no parallelism. Spamc keeps getting called, one message at a time. There is noone else running any mail service, nothing else calling on spamd. Only me. And the procmail gets the result from spamc, spamc has exited, procmail delivers the message and goes on to the next one. So why have many spamd children? I can see the logic in having one forked off ready, but having 4? Having 8? Yes that would make sense in a richly multi-user context where there are going to be many users/threads calling on it in parallel. But that doesn't apply for my context. And I think many other users are signle users of their machines. -- "Of all tyrannies a tyranny sincerely exercised for the good of its victims may be the most oppressive. It may be better to live under robber barons than under omnipotent moral busybodies, the robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for own good will torment us without end, for they do so with the approval of their own conscience." -- C.S. Lewis, _God in the Dock_ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 05/07/2014 12:37 AM, Per Jessen wrote:
Carlos E. R. wrote:
On Monday, 2014-05-05 at 12:29 -0700, Linda Walsh wrote:
** Significantly: "--max-conn-per-child=1"
It is not in the spamd manual.
It is in mine, but my spamassassin is backlevel and built from source:
NAME spamd - daemonized version of spamassassin
SYNOPSIS spamd [options]
Options:
[snip] -m num, --max-children=num Allow maximum num children --min-children=num Allow minimum num children --min-spare=num Lower limit for number of spare children --max-spare=num Upper limit for number of spare children --max-conn-per-child=num Maximum connections accepted by child before it is respawned
So what's the significance of this?
Like many others here I run a single user system, its just me who who uses it.
I bring in my mail using fetchmail which feeds into procmail. The procmail makes use of spamc, the client that calls on spamd.
It processes one mail message at a time. I can see that by watching the logs with 'tail -f'.
I emphasise: ONE MESSAGE AT A TIME.
[snip]
So why have many spamd children?
I can see the logic in having one forked off ready, but having 4? Having 8? Yes that would make sense in a richly multi-user context where there are going to be many users/threads calling on it in parallel. But that doesn't apply for my context.
If those are the default, just change them to suit your context :-) Anton, I'm sure you're making a point here, but I can't see what it is? -- Per Jessen, Zürich (14.9°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/07/2014 08:38 AM, Per Jessen wrote:
Anton, I'm sure you're making a point here, but I can't see what it is?
Same one I always make :-) Context is Everything -- Things are more like they are now than they ever were before. -- Dwight D. Eisenhower -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/07/2014 08:38 AM, Per Jessen wrote:
If those are the default, just change them to suit your context :-)
I've tried setting 'max-children" in /etc/sysconfig/spamd to "1" and "2" and both cases spamd failed when restarted with systemctl. I've gone back to the defaaults, which seems to spawn two children. -- Men fear thought as they fear nothing else on earth, more than ruin, more even than death. . . . Thought is subversive and revolutionary, destructive and terrible, thought is merciless to privilege, established institutions, and comfortable habit. Thought looks into the pit of hell and is not afraid. Thought is great and swift and free, the light of the world, and the chief glory of man. -- Bertrand Russell -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-05-07 14:01, Anton Aylward wrote:
So what's the significance of this?
Like many others here I run a single user system, its just me who who uses it.
Like me.
I bring in my mail using fetchmail which feeds into procmail. The procmail makes use of spamc, the client that calls on spamd.
My fetchmail sends to postfix, who eventually calls procmail. And it can send several posts at the same time. If the email is processed by procmail faster than fetchmail gets them, it goes one by one. If not, it starts sending more simultaneously, up to a limit defined in "/etc/postfix/master.cf": procmail unix - n n - 7 pipe flags=R user=nobody argv=/usr/bin/procmail -t -m /etc/procmailrc ${sender} ${recipient} 7 is the limit in my case, and they do get used. And I need more than one, because each email takes time to process; currently: spamd: clean message (-2.7/5.0) for cer:1000 in 2.7 seconds, 6835 bytes. spamd: clean message (-2.6/5.0) for cer:1000 in 3.7 seconds, 8887 bytes. spamd: clean message (0.4/5.0) for cer:1000 in 3.5 seconds, 8195 bytes. spamd: clean message (-2.6/5.0) for cer:1000 in 3.7 seconds, 7195 bytes. spamd: clean message (-1.6/5.0) for cer:1000 in 3.4 seconds, 11125 bytes. Some take longer: spamd: clean message (-101.9/5.0) for cer:1000 in 9.7 seconds, 3561 bytes. Some way longer: spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes. (see the size) or spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by At times, I got consistent processing times of 6..12 seconds each. A hundred emails would be processed in 1200 seconds, which I find intolerable - with 5% CPU load at most. The hurdle were the network test timeouts, I think. So yes, spamd with fetchmail paralelizes - if you put postfix in the equation. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
Carlos E. R. wrote:
Some way longer:
spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes.
(see the size)
or
spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by
That's a very large spam-mail - quite unusual. 213 seconds to process it could be caused by lots of DNS timeouts.
At times, I got consistent processing times of 6..12 seconds each. A hundred emails would be processed in 1200 seconds, which I find intolerable - with 5% CPU load at most. The hurdle were the network test timeouts, I think.
Yes, almost certainly. -- Per Jessen, Zürich (15.6°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-05-07 15:35, Per Jessen wrote:
Carlos E. R. wrote:
Some way longer:
spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes.
(see the size)
or
spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by
That's a very large spam-mail - quite unusual. 213 seconds to process it could be caused by lots of DNS timeouts.
But it is not spam :-) It was sent by me (and to me), from my android phone. It is a dump of just one WhatsApp chat. Yes, it could be DNS timeouts. I have the vague idea that cpu load was high, but I'm unsure. I would have to force filter it again with spamd - ah, I know how. Send to another local account.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13292 cer2 20 0 390280 296056 2112 R 99,87 3,613 0:20.82 spamd child
So, it is not DNS timeouts, it is really processing load.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13292 cer2 20 0 390280 296056 2112 R 99,85 3,613 2:39.40 spamd child
It finished:
<2.6> 2014-05-07 17:19:23 Telcontar spamd 13292 - - spamd: clean message (4.3/5.0) for cer2:1012 in 172.9 seconds, 15982025 bytes. <2.6> 2014-05-07 17:19:23 Telcontar spamd 13292 - - spamd: result: . 4 - DRUGS_MUSCLE,FREEMAIL_FROM,FUZZY_XPILL,LOTS_OF_MONEY,NORMAL_HTTP_TO_IP,T_FRT_COCK,T_FRT_FUCK1,WEIRD_QUOTING scantime=172.9,size=15982025,user=cer2,uid=1012,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=46429,mid=<alpine.LSU.2.11.1405071712370.5918@Telcontar.valinor>,autolearn=no <2.6> 2014-05-07 17:19:23 Telcontar spamd 13289 - - prefork: child states: BI <2.6> 2014-05-07 17:19:23 Telcontar spamd 13289 - - spamd: handled cleanup of child pid [13292] due to SIGCHLD: exit 0 <2.6> 2014-05-07 17:19:23 Telcontar spamd 13289 - - spamd: server successfully spawned child process, pid 10121 <2.6> 2014-05-07 17:19:23 Telcontar spamd 13289 - - prefork: child states: II <2.6> 2014-05-07 17:19:23 Telcontar postfix 10015 - - 27ED861664: to=<cer2@localhost.valinor>, relay=local, delay=174, delays=0.43/0/0/173, dsn=2.0.0, status=sent (delivered to command: /usr/bin/procmail) <2.6> 2014-05-07 17:19:23 Telcontar postfix 3805 - - 27ED861664: removed
It went a bit faster than the other time, /just/ 173 seconds. And this time it was thought clean. Go figure. At least the child was instantly respawned, as I had changed the config, just one child per email. And memory size did not go that up - or it was not that precise email, there were several similar posts sent in a batch. Maybe I have to send several, locally, and remove --max-conn-per-child=1 temporarily, to reproduce the problem. Doable. Coffee first. :-) -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On 05/07/2014 11:36 AM, Carlos E. R. wrote:
But it is not spam :-)
It was sent by me (and to me), from my android phone. It is a dump of just one WhatsApp chat.
I make aggressive use of whitelisting and blacklisting in my procmail BEFORE submitting to spamassassin, even though spamassassin config also has its own whitelist and blacklist. If I submit from my android sent by me to me it will never go though spamassassin. Yes this does take a bit more checking to make sure it is not being spoofed, but procmail calling formmail and grep is cheaper than spamassassin. It would be more so for you if you are calling upon a outside RBL service and the network timeout. -- Give a man a fish, and you'll feed him for a day. Give him a religion, and he'll starve to death while praying for a fish. --Timothy Jones -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-05-07 17:51, Anton Aylward wrote:
On 05/07/2014 11:36 AM, Carlos E. R. wrote:
But it is not spam :-)
It was sent by me (and to me), from my android phone. It is a dump of just one WhatsApp chat.
I make aggressive use of whitelisting and blacklisting in my procmail BEFORE submitting to spamassassin, even though spamassassin config also has its own whitelist and blacklist.
If I submit from my android sent by me to me it will never go though spamassassin.
Yes this does take a bit more checking to make sure it is not being spoofed, but procmail calling formmail and grep is cheaper than spamassassin. It would be more so for you if you are calling upon a outside RBL service and the network timeout.
It is something to consider, thanks. But takes time and some work to do... -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On 05/07/2014 06:22 PM, Carlos E. R. wrote:
On 2014-05-07 17:51, Anton Aylward wrote:
On 05/07/2014 11:36 AM, Carlos E. R. wrote:
But it is not spam :-)
It was sent by me (and to me), from my android phone. It is a dump of just one WhatsApp chat.
I make aggressive use of whitelisting and blacklisting in my procmail BEFORE submitting to spamassassin, even though spamassassin config also has its own whitelist and blacklist.
If I submit from my android sent by me to me it will never go though spamassassin.
Yes this does take a bit more checking to make sure it is not being spoofed, but procmail calling formmail and grep is cheaper than spamassassin. It would be more so for you if you are calling upon a outside RBL service and the network timeout.
It is something to consider, thanks. But takes time and some work to do...
IIR I found the recipes on the 'net. Feed though formail and then grep. Actually the whitelisting, which is a lot easier, probably does more to take the load off spamassassin than the blacklisting. I've whitelisted my common correspondents and the lists I subscribe to: :::::::::::::: whitelist.rc :::::::::::::: # Test if the email's sender is whitelisted; if so, send it straight to # $DEFAULT. Note that this comes before any other filters. :0: * ? formail -x"From" -x"From:" -x"Sender:" \ -x"Reply-To:" -x"Return-Path:" -x"To:" \ | egrep -is -f ${HOME}/.whitelist { LOG="Found on whitelist$NL" $DEFAULT }
From what you've supplied so far, Carlos, this and few other simple recipies that are available on the net, for example to deal with attachments over a certain size, can make things a lot easier. You might also consider recipes that filter by attachment type.
http://pm-doc.sourceforge.net/doc/#procmail_mime_and_html -- "A Meltdown? One of those annoying buzzwords. We prefer to think of it as an unrequested fission surplus!" -- Monty Burns -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
Actually the whitelisting, which is a lot easier, probably does more to take the load off spamassassin than the blacklisting. I've whitelisted my common correspondents and the lists I subscribe to:
:::::::::::::: whitelist.rc ::::::::::::::
# Test if the email's sender is whitelisted; if so, send it straight # to $DEFAULT. Note that this comes before any other filters. :0: * ? formail -x"From" -x"From:" -x"Sender:" \ -x"Reply-To:" -x"Return-Path:" -x"To:" \ | egrep -is -f ${HOME}/.whitelist
I'm sure that works very well for you, but because email-addresses are so easily forged, using the SA whitelisting directives: whitelist_from_rcvd, whitelist_from_spf, or whitelist_from_dkim would be "safer". Just FYI. -- Per Jessen, Zürich (11.0°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/08/2014 01:42 AM, Per Jessen wrote:
Anton Aylward wrote:
Actually the whitelisting, which is a lot easier, probably does more to take the load off spamassassin than the blacklisting. I've whitelisted my common correspondents and the lists I subscribe to:
:::::::::::::: whitelist.rc ::::::::::::::
# Test if the email's sender is whitelisted; if so, send it straight # to $DEFAULT. Note that this comes before any other filters. :0: * ? formail -x"From" -x"From:" -x"Sender:" \ -x"Reply-To:" -x"Return-Path:" -x"To:" \ | egrep -is -f ${HOME}/.whitelist
I'm sure that works very well for you, but because email-addresses are so easily forged, using the SA whitelisting directives:
whitelist_from_rcvd, whitelist_from_spf, or whitelist_from_dkim
would be "safer". Just FYI.
Perfectly true and I do use SA's whitelisting and blacklisting features. But the small set handled this way represents a lot of very personal traffic and is easily managed., just editing a text file. No need to restart SA to get it to re-read or recompile a rule set. Formail and grep forked from procmail might nt be as cheap as if I implemented the whitelist in procmails own RE, but the text file is easier to deal with. Either is cheaper than launching spamassassin. I'm sorry if I gave the impression that this has to be an absolutely either/or situation, Its not. Its not as if I rely solely on whitelisting. Its about taking the load off a very heavily tuned spamassassin. If you happen to know the addresses of any of my relatives to forge their address then you'll find there is additional procmail and thunderbird filtering :-) Suppose a 'false negative' does get though. So? I get a couple of false negatives getting past SA each week. That's why we have 'sa-learn'. The spammers keep coming up with new formats and we have to keep up as well. -- Public confidence in the integrity of the Government is indispensable to faith in democracy; and when we lose faith in the system, we have lost faith in everything we fight and spend for. Adlai E. Stevenson Jr. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On 2014-05-07 15:35, Per Jessen wrote:
Carlos E. R. wrote:
Some way longer:
spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes.
(see the size)
or
spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by
That's a very large spam-mail - quite unusual. 213 seconds to process it could be caused by lots of DNS timeouts.
But it is not spam :-)
According to spamassasin it was :-)
Yes, it could be DNS timeouts. I have the vague idea that cpu load was high, but I'm unsure. I would have to force filter it again with spamd - ah, I know how. Send to another local account.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13292 cer2 20 0 390280 296056 2112 R 99,87 3,613 0:20.82 spamd child
So, it is not DNS timeouts, it is really processing load.
Then it's simply due to the size of it. SA is generally speaking a large collection of regex'es, they're totally CPU-bound.
It went a bit faster than the other time, /just/ 173 seconds.
And this time it was thought clean. Go figure.
If you want to know, look at the rules that were hit. -- Per Jessen, Zürich (14.5°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-05-07 20:44, Per Jessen wrote:
Carlos E. R. wrote:
But it is not spam :-)
According to spamassasin it was :-)
Ok, right :-)
Then it's simply due to the size of it. SA is generally speaking a large collection of regex'es, they're totally CPU-bound.
It went a bit faster than the other time, /just/ 173 seconds.
And this time it was thought clean. Go figure.
If you want to know, look at the rules that were hit.
Some other day :-) -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On 2014-05-07 17:36, Carlos E. R. wrote:
On 2014-05-07 15:35, Per Jessen wrote:
Maybe I have to send several, locally, and remove --max-conn-per-child=1 temporarily, to reproduce the problem. Doable.
Coffee first. :-)
I did send a bunch, and the sizes went up. At the end, several children were killed, but two remained, one .4 GB, another .6 GB (RSS is ps output). The solution would be to not have any child pre-spawned, or allow only one connection per child. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
Carlos E. R. wrote:
On 2014-05-07 17:36, Carlos E. R. wrote:
On 2014-05-07 15:35, Per Jessen wrote:
Maybe I have to send several, locally, and remove --max-conn-per-child=1 temporarily, to reproduce the problem. Doable.
Coffee first. :-)
I did send a bunch, and the sizes went up. At the end, several children were killed, but two remained, one .4 GB, another .6 GB (RSS is ps output).
The solution would be to not have any child pre-spawned, or allow only one connection per child.
Not be too picky, but that's only a work-around, not a solution. Anyway, sounds like you've got a reproducable problem, good going. I'm sure the SA guys will want to know. -- Per Jessen, Zürich (11.2°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-05-08 07:32, Per Jessen wrote:
Carlos E. R. wrote:
Not be too picky, but that's only a work-around, not a solution. Anyway, sounds like you've got a reproducable problem, good going. I'm sure the SA guys will want to know.
It will have to wait. I have to hit the road. But I doubt they don't know that children do not release memory. I think then that the parent should watch that, and respawn children automatically when they grow a lot. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
Carlos E. R. wrote:
On 2014-05-08 07:32, Per Jessen wrote:
Carlos E. R. wrote:
Not be too picky, but that's only a work-around, not a solution. Anyway, sounds like you've got a reproducable problem, good going. I'm sure the SA guys will want to know.
It will have to wait. I have to hit the road.
But I doubt they don't know that children do not release memory.
Yes, but that is not your problem right? the problem you're seeing is that children aren't being respawned when they should be. That's how I understood it.
I think then that the parent should watch that, and respawn children automatically when they grow a lot.
"grow a lot" is, as Anton likes to point out, context-sensitive. Besides, SA has a work-around already - respawn after X mails. -- Per Jessen, Zürich (17.9°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/07/2014 09:21 AM, Carlos E. R. wrote:
Some way longer:
spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes.
(see the size)
or
spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by
At times, I got consistent processing times of 6..12 seconds each. A hundred emails would be processed in 1200 seconds, which I find intolerable - with 5% CPU load at most. The hurdle were the network test timeouts, I think.
WOW! I process about 200 messages a day and grepping though the logs I see one at 7 seconds, two at 5 second and the rest at 1-2 seconds. My load average over that period is less than 0.5 (It occasionally peaked when I was running the nvidia driver from nvidia! Burt that's a different matter) This is with a single core CPU: Model name: AMD Athlon(tm) 64 Processor 3000+ Stepping: 2 CPU MHz: 1889.930 BogoMIPS: 3779.86 Perhaps you are right about the network - Razr? - timeouts. I have that disabled.
So yes, spamd with fetchmail paralelizes - if you put postfix in the equation.
Unless you have a very quick description of the mail.cf as well as the master.cf lets discuss that off-line. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 05/07/2014 09:21 AM, Carlos E. R. wrote:
Some way longer:
spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes.
(see the size)
or
spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by
At times, I got consistent processing times of 6..12 seconds each. A hundred emails would be processed in 1200 seconds, which I find intolerable - with 5% CPU load at most. The hurdle were the network test timeouts, I think.
WOW!
I process about 200 messages a day and grepping though the logs I see one at 7 seconds, two at 5 second and the rest at 1-2 seconds.
My load average over that period is less than 0.5
(It occasionally peaked when I was running the nvidia driver from nvidia! Burt that's a different matter)
This is with a single core CPU: Model name: AMD Athlon(tm) 64 Processor 3000+ Stepping: 2 CPU MHz: 1889.930 BogoMIPS: 3779.86
Yeah, the CPU is largely irrelevant - I run my personal mail through our test-system cluster which is comprised of four Pentium II, 400MHz .... single-core :-) Scan-time is typically about 20 seconds, with exceptions to either side. -- Per Jessen, Zürich (15.0°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
Anton Aylward wrote:
On 05/07/2014 09:21 AM, Carlos E. R. wrote:
Some way longer:
spamd: clean message (-2.7/5.0) for cer:1000 in 57.3 seconds, 5630 spamd: clean message (-2.6/5.0) for cer:1000 in 53.2 seconds, 7839 bytes.
(see the size)
or
spamd: identified spam (6.5/5.0) for cer:1000 in 213.1 seconds, 15976538 by
At times, I got consistent processing times of 6..12 seconds each. A hundred emails would be processed in 1200 seconds, which I find intolerable - with 5% CPU load at most. The hurdle were the network test timeouts, I think.
WOW!
I process about 200 messages a day and grepping though the logs I see one at 7 seconds, two at 5 second and the rest at 1-2 seconds.
My load average over that period is less than 0.5
(It occasionally peaked when I was running the nvidia driver from nvidia! Burt that's a different matter)
This is with a single core CPU: Model name: AMD Athlon(tm) 64 Processor 3000+ Stepping: 2 CPU MHz: 1889.930 BogoMIPS: 3779.86
Yeah, the CPU is largely irrelevant
Except for people scanning excessively large emails of several megabytes :-) -- Per Jessen, Zürich (14.0°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/07/2014 02:48 PM, Per Jessen wrote:
Yeah, the CPU is largely irrelevant Except for people scanning excessively large emails of several megabytes :-)
I don't know about you, but in my .procmailrc I have a rule that limits how large an email file should be submitted to spamassassin: # Pipe the mail through spamassassin # (replace 'spamassassin' with 'spamc' # if you use the spamc/spamd combination) # # The condition line ensures that only messages smaller than 250 kB # (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam # isn't bigger than a few k and working with big messages can bring # SpamAssassin to its knees. # # The lock file ensures that only 1 spamassassin invocation happens # at 1 time, to keep the load down. # :0 fw: /tmp/spamassassin.lock * < 256000 | /usr/bin/spamc -- The spirit of resistance to government is so valuable on certain occasions that I wish it to be always kept alive. It will often be exercised when wrong, but better so than not to be exercised at all. --Thomas Jefferson -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-05-07 23:54, Anton Aylward wrote:
On 05/07/2014 02:48 PM, Per Jessen wrote:
Yeah, the CPU is largely irrelevant Except for people scanning excessively large emails of several megabytes :-)
I don't know about you, but in my .procmailrc I have a rule that limits how large an email file should be submitted to spamassassin:
As I do, but I get large spam emails. I just saw one of 20 MB, but many are over 400KB (a hundred at least are in my training folder). With a low limit, they pass through. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
Carlos E. R. wrote:
On 2014-05-07 23:54, Anton Aylward wrote:
On 05/07/2014 02:48 PM, Per Jessen wrote:
Yeah, the CPU is largely irrelevant Except for people scanning excessively large emails of several megabytes :-)
I don't know about you, but in my .procmailrc I have a rule that limits how large an email file should be submitted to spamassassin:
As I do, but I get large spam emails. I just saw one of 20 MB, but many are over 400KB (a hundred at least are in my training folder). With a low limit, they pass through.
20Mb is really unusual - it's simply not efficient for a spammer to be sending such big spams. In the same time it takes to send 20Mb, he could send 200 mails of size 100Kb. -- Per Jessen, Zürich (11.1°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 05/07/2014 02:48 PM, Per Jessen wrote:
Yeah, the CPU is largely irrelevant Except for people scanning excessively large emails of several megabytes :-)
I don't know about you, but in my .procmailrc I have a rule that limits how large an email file should be submitted to spamassassin:
Alternatively, use the spamc '-s' option to the max size to be processed. (the end effect is the same of course). -- Per Jessen, Zürich (11.2°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/08/2014 01:30 AM, Per Jessen wrote:
Anton Aylward wrote:
On 05/07/2014 02:48 PM, Per Jessen wrote:
Yeah, the CPU is largely irrelevant Except for people scanning excessively large emails of several megabytes :-)
I don't know about you, but in my .procmailrc I have a rule that limits how large an email file should be submitted to spamassassin:
Alternatively, use the spamc '-s' option to the max size to be processed. (the end effect is the same of course).
... For some values of "same". The test I describe is done entirely within procmail and does not require forking spamc in the first place. While fork/execl may be much cheaper with *NIX than VMS, which was the whole point back when, its still cheaper not to do it. You are making it very clear that the 20MB files are crippling large for SA and not a cost effective strategy for a spammer. If its a case of sending such as an attachment of 'dump' from you cell phone to your desktop than this is a case of source-based whitelisting to bypass SA. That's easy to set up as a recipe in procmail. Part of the analysis should be 'how often does this happen?' If you only send such data dumps to yourself one in a while then its not worth the hassle. If this happens every day or every week then yes, do something about it. -- "Key escrow to rule them all; key escrow to find them. Key escrow to bring them all and in the darkness bind them. In the land of surveillance where Big Brother lies." -- Peter Gutmann -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
So why have many spamd children?
I can see the logic in having one forked off ready, but having 4? Having 8? Yes that would make sense in a richly multi-user context where there are going to be many users/threads calling on it in parallel. But that doesn't apply for my context.
And I think many other users are signle users of their machines.
The only time my spamd needs that many is after my 'fetchmail' process has been down a while (system down, or rebooted, and didn't restart fetchmail right away, or detected some other problem and killed the fetcher...etc.). Spamd takes about 7-10 seconds/email. But most of that time is *waiting* (network lookups). So, right after a restart of I might have a couple of hundred messages come in (have processed as many as 500-800 at once). They still all get processed in about a minute, maybe two at most. Sure, it could process those, say 700 emails sequentially, and take up to 2 hours to process the queue. BUT if it can process a few hundred in parallel, why not let it go? Does that answer your Q? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-05-07 06:37, Per Jessen wrote:
Carlos E. R. wrote:
On Monday, 2014-05-05 at 12:29 -0700, Linda Walsh wrote:
** Significantly: "--max-conn-per-child=1"
It is not in the spamd manual.
It is in mine, but my spamassassin is backlevel and built from source:
Then the openSUSE package has the wrong manuals. :-? -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
Carlos E. R. wrote:
On 2014-05-07 06:37, Per Jessen wrote:
Carlos E. R. wrote:
On Monday, 2014-05-05 at 12:29 -0700, Linda Walsh wrote:
** Significantly: "--max-conn-per-child=1"
It is not in the spamd manual.
It is in mine, but my spamassassin is backlevel and built from source:
Then the openSUSE package has the wrong manuals. :-?
Yes, that's what it sounds like. -- Per Jessen, Zürich (14.9°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
El 05/05/14 05:56, Carlos E. R. escribió:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4034 root 20 0 1295260 1,146g 2780 S 0,000 14,66 8:07.59 spamd child 4035 root 20 0 1206876 1,061g 2908 S 0,000 13,58 4:09.77 spamd child
1 gigabyte resident size?
spamd is a perl program right ? it might be expected that it does not free memory. I do not know if perl frees memory only at exit, has a garbage collector or a combination of both. - -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/05/2014 05:08 PM, Cristian Rodríguez wrote:
spamd is a perl program right ? it might be expected that it does not free memory. I do not know if perl frees memory only at exit, has a garbage collector or a combination of both.
With perl there is the interpreter written in C and the script which is a text file, and then there is the data-that-is-data. I realise also that there are some plugins that are compiled C code. So what is being reported here? As far as the interpreter itself goes, I expect it will be treated like any other compiled program with a core 'working set'. That code will be shared with any other instance of perl and with the user-context-client that is spawned to process a client's mail message. That client copy will process the mail message as data-that-is-data, and if I read correctly with exit when done. Strictly speaking there only needs to be one copy of spamd the parent process. :-) HA HA. See how Apache works and wonder how many 'parent' copies you want to be able to deal with various incoming mail streams. I'm the only user of my system, and mail comes in via fetchmail+procmail and procmail invoking spamc, only one source at a time, so having only one parent process and only one child at a time (-max-children) is all I need. YMMV. -- To do anything in this world worth doing, we must not stand back shivering and thinking of the cold and danger, but jump in, and scramble through as well as we can. Sydney Smith -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 05/05/2014 05:08 PM, Cristian Rodríguez wrote:
spamd is a perl program right ? it might be expected that it does not free memory. I do not know if perl frees memory only at exit, has a garbage collector or a combination of both.
With perl there is the interpreter written in C and the script which is a text file, and then there is the data-that-is-data. I realise also that there are some plugins that are compiled C code.
So what is being reported here?
As far as the interpreter itself goes, I expect it will be treated like any other compiled program with a core 'working set'. That code will be shared with any other instance of perl and with the user-context-client that is spawned to process a client's mail message.
That client copy will process the mail message as data-that-is-data, and if I read correctly with exit when done.
No, that's not quite how spamd works. It forks a number of child processes that will process a number of messages before being re-spawned. -- Per Jessen, Zürich (14.3°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2014-05-06 at 00:26 -0400, Anton Aylward wrote:
Strictly speaking there only needs to be one copy of spamd the parent process. :-) HA HA. See how Apache works and wonder how many 'parent' copies you want to be able to deal with various incoming mail streams. I'm the only user of my system, and mail comes in via fetchmail+procmail and procmail invoking spamc, only one source at a time, so having only one parent process and only one child at a time (-max-children) is all I need. YMMV.
There is one parent, and several children when needed, up to a maximum of 8. I allow several children because some of the network based tests seem to need up to 6 seconds, as spammed waits for an answer and doing nothing all that time. So, when processing a bunch of email from this email list, say 50, it took a long time to do. Having several children waiting speeds things a lot. - -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlNpXTUACgkQtTMYHG2NR9V/XACfQNRa60unhtzuzt5x192UoGXY QiYAoIw8mC5JcTcBqfeS3Qr71WWz3pDd =z+tz -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4034 root 20 0 1295260 1,146g 2780 S 0,000 14,66 8:07.59 spamd child 4035 root 20 0 1206876 1,061g 2908 S 0,000 13,58 4:09.77 spamd child
1 gigabyte resident size?
That seems pretty unusual. I run a test-system cluster with 4 very old boxes, each with only 384Mb RAM. They each run typically 2-3 spamd children, but don't think I've ever seen that kind of memory usage. Mind you, I only scan mails that are less than 2Mb.
Ah, and that email is not spam. It is a dump of a WhatsApp chat with multimedia attachments.
How big is the email itself? spamd does need to keep it in-core, so until a child is re-spawned, the memory usage might remain at the amount required for the largest email. -- Per Jessen, Zürich (14.4°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2014-05-06 at 09:15 +0200, Per Jessen wrote:
Carlos E. R. wrote:
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4034 root 20 0 1295260 1,146g 2780 S 0,000 14,66 8:07.59 spamd child 4035 root 20 0 1206876 1,061g 2908 S 0,000 13,58 4:09.77 spamd child
1 gigabyte resident size?
That seems pretty unusual. I run a test-system cluster with 4 very old boxes, each with only 384Mb RAM. They each run typically 2-3 spamd children, but don't think I've ever seen that kind of memory usage. Mind you, I only scan mails that are less than 2Mb.
I increased that limit, because I often got spam mails way bigger than that, that went through to my inbox.
Ah, and that email is not spam. It is a dump of a WhatsApp chat with multimedia attachments.
How big is the email itself?
"Just" 19 MB. Not that /huge/.
spamd does need to keep it in-core, so until a child is re-spawned, the memory usage might remain at the amount required for the largest email.
The huge memory usage was by the children, not the parent. That email contains about a hundred attachments. They are marked like: 25 47 KB Video/3GPP (Name: "PTT-20130807-WA0018.3gp") 26 ~10 KB Text/* (Name: "PTT-20130807-WA0019.3ga") 27 89 KB Image/JPEG (Name: "IMG-20130807-WA0020.jpg") 28 74 KB Image/JPEG (Name: "IMG-20130808-WA0000.jpg") 29 ~17 KB Text/* (Name: "PTT-20130808-WA0001.3ga") 30 ~20 KB Text/* (Name: "PTT-20130808-WA0002.3ga") 31 ~10 KB Text/* (Name: "PTT-20130808-WA0003.3ga") Notice that the .3ga are marked text, when they are multimedia (voice messages, I think). Thunderbird tries to display them, and obviously fails. WhatsApp has made a mess of it. I have a /etc/sysconfig/spamd with these settings (old): SPAMD_AWL=no SPAMD_NICE=yes SPAMD_ARGS="-d -c --max-children=8 " SPAM_SA_UPDATE="no" SPAM_SA_COMPILE="no" SPAM_AMAVISD_RESTART="no" SPAM_SPAMD_RESTART="yes" But there is also a "spampd" file with this single line: SPAMPD_OPTIONS="--port=10025 --relayhost=127.0.0.1:10026 --user=spamfilter --tagall --children=5 --maxsize=250" which are some of the options that Linda has, it looks like. But: cer@Telcontar:/etc/sysconfig> rpm -qf /etc/sysconfig/spampd file /etc/sysconfig/spampd is not owned by any package cer@Telcontar:/etc/sysconfig> It is dated "Feb 25 21:45". I don't know what "SPAMPD" is. - -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlNpYysACgkQtTMYHG2NR9WH3gCeMuIF3vxDDW+JNaNjGNDXt40l XdAAn10/kB3tiaUVREVsryV85abx4xns =BzmK -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tuesday, 2014-05-06 at 09:15 +0200, Per Jessen wrote:
Carlos E. R. wrote:
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4034 root 20 0 1295260 1,146g 2780 S 0,000 14,66 8:07.59 spamd child 4035 root 20 0 1206876 1,061g 2908 S 0,000 13,58 4:09.77 spamd child
1 gigabyte resident size?
That seems pretty unusual. I run a test-system cluster with 4 very old boxes, each with only 384Mb RAM. They each run typically 2-3 spamd children, but don't think I've ever seen that kind of memory usage. Mind you, I only scan mails that are less than 2Mb.
I increased that limit, because I often got spam mails way bigger than that, that went through to my inbox.
Ah, and that email is not spam. It is a dump of a WhatsApp chat with multimedia attachments.
How big is the email itself?
"Just" 19 MB. Not that /huge/.
Any chance previous emails might have been much bigger? The number of emails processed by spamd before it is respawned is set with "--max-conn-per-child". Maybe you can find a much bigger mail in the log?
spamd does need to keep it in-core, so until a child is re-spawned, the memory usage might remain at the amount required for the largest email.
The huge memory usage was by the children, not the parent.
Yes, that is to be expected - the parent only accepts the connection and passes it to the child, it doesn't do any work itself.
That email contains about a hundred attachments.
Okay, that is pretty unusual, I would say - maybe you could reproduce the problem with another such email? If spamassassin has some sort of ballooning memory issue when processing mails with many attachments, I'm sure the developers would like to know.
I have a /etc/sysconfig/spamd with these settings (old):
SPAMD_AWL=no SPAMD_NICE=yes SPAMD_ARGS="-d -c --max-children=8 " SPAM_SA_UPDATE="no" SPAM_SA_COMPILE="no" SPAM_AMAVISD_RESTART="no" SPAM_SPAMD_RESTART="yes"
But there is also a "spampd" file with this single line:
SPAMPD_OPTIONS="--port=10025 --relayhost=127.0.0.1:10026 --user=spamfilter --tagall --children=5 --maxsize=250"
Those are clearly options for the spamd daemon. If you google "SPAMPD_OPTIONS" you'll see a few hits here and there. -- Per Jessen, Zürich (12.8°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-05-07 06:33, Per Jessen wrote:
Carlos E. R. wrote:
How big is the email itself?
"Just" 19 MB. Not that /huge/.
Any chance previous emails might have been much bigger? The number of emails processed by spamd before it is respawned is set with "--max-conn-per-child". Maybe you can find a much bigger mail in the log?
No, I just looked. Typically, when I fetch email the number of childs increase, and at the end of activity it drops to just two. It just chanced that the two were those emails, because they took several minutes to process.
That email contains about a hundred attachments.
Okay, that is pretty unusual, I would say - maybe you could reproduce the problem with another such email? If spamassassin has some sort of ballooning memory issue when processing mails with many attachments, I'm sure the developers would like to know.
I can send it to myself, internally. It would get processed. [...] Nope, it did not, just tried. I do not know why :-? Ah, yes, I do, the local account is not filtered. However, I can not provide this sample post to anyone, it is private.
I have a /etc/sysconfig/spamd with these settings (old):
SPAMD_AWL=no SPAMD_NICE=yes SPAMD_ARGS="-d -c --max-children=8 " SPAM_SA_UPDATE="no" SPAM_SA_COMPILE="no" SPAM_AMAVISD_RESTART="no" SPAM_SPAMD_RESTART="yes"
But there is also a "spampd" file with this single line:
SPAMPD_OPTIONS="--port=10025 --relayhost=127.0.0.1:10026 --user=spamfilter --tagall --children=5 --maxsize=250"
Those are clearly options for the spamd daemon. If you google "SPAMPD_OPTIONS" you'll see a few hits here and there.
But they are not used, as seen in the "ps" output: root 13289 0.0 0.8 151520 66388 ? Ss 00:30 0:05 /usr/sbin/spamd -d -c --max-children=8 --max-conn-per-child=1 -r /var/run/spamd.pid I added the max-conn-per-child yesterday. But notice the max-children=8, not 5. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
Carlos E. R. wrote:
On 2014-05-07 06:33, Per Jessen wrote:
Carlos E. R. wrote:
How big is the email itself?
"Just" 19 MB. Not that /huge/.
Any chance previous emails might have been much bigger? The number of emails processed by spamd before it is respawned is set with "--max-conn-per-child". Maybe you can find a much bigger mail in the log?
No, I just looked.
Typically, when I fetch email the number of childs increase, and at the end of activity it drops to just two. It just chanced that the two were those emails, because they took several minutes to process.
That email contains about a hundred attachments.
Okay, that is pretty unusual, I would say - maybe you could reproduce the problem with another such email? If spamassassin has some sort of ballooning memory issue when processing mails with many attachments, I'm sure the developers would like to know.
I can send it to myself, internally. It would get processed. [...] Nope, it did not, just tried. I do not know why :-? Ah, yes, I do, the local account is not filtered.
However, I can not provide this sample post to anyone, it is private.
To quickly test if the many attachments is the issue, you could just send yourself another mail with a lot of attachments. -- Per Jessen, Zürich (14.1°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-05-07 15:17, Per Jessen wrote:
Carlos E. R. wrote:
To quickly test if the many attachments is the issue, you could just send yourself another mail with a lot of attachments.
I have first to modify first procmail so that local email is processed by spamd. It may be the many attachments, but it can also be the type of those. Notice that whatsapp classifies as plain text the voice messages, so spamd might try to analyse that "text". But the issue, to me, is not really the huge memory, but that the child was not restarted after processing, if memory can not be freed. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
Carlos E. R. wrote:
On 2014-05-07 15:17, Per Jessen wrote:
Carlos E. R. wrote:
To quickly test if the many attachments is the issue, you could just send yourself another mail with a lot of attachments.
I have first to modify first procmail so that local email is processed by spamd. It may be the many attachments, but it can also be the type of those. Notice that whatsapp classifies as plain text the voice messages, so spamd might try to analyse that "text".
But the issue, to me, is not really the huge memory, but that the child was not restarted after processing, if memory can not be freed.
Ah okay - I didn't realise that was the issue. That sounds like something to report to the spamassassin people. -- Per Jessen, Zürich (15.9°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-05-07 16:55, Per Jessen wrote:
Carlos E. R. wrote:
But the issue, to me, is not really the huge memory, but that the child was not restarted after processing, if memory can not be freed.
Ah okay - I didn't realise that was the issue. That sounds like something to report to the spamassassin people.
Yes, memory is there to be used as needed, as long as it is released when not. The max-conn-per-child switch handles that part, I think. Maybe that causes some performance decrease, spawning more often - but Linux is good at that. I can do some limited testing now, I have to prepare for a little field trip. Reporting takes longer. And if the openSUSE package is different than the upstream one (the obsolete man page indicates that), I can't report to them, but to openSUSE instead. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
Carlos E. R. wrote:
On 2014-05-07 16:55, Per Jessen wrote:
Carlos E. R. wrote:
But the issue, to me, is not really the huge memory, but that the child was not restarted after processing, if memory can not be freed.
Ah okay - I didn't realise that was the issue. That sounds like something to report to the spamassassin people.
Yes, memory is there to be used as needed, as long as it is released when not.
The max-conn-per-child switch handles that part, I think. Maybe that causes some performance decrease, spawning more often - but Linux is good at that.
And as you are not running a high-performance systems with thousands of users, it hardly matters anyway.
And if the openSUSE package is different than the upstream one (the obsolete man page indicates that), I can't report to them, but to openSUSE instead.
Nah, as long as the source version is the same, and the issue is reproducable, go ahead and talk to the SA guys. -- Per Jessen, Zürich (14.9°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, 7 May 2014 15:02:59 Carlos E. R. wrote:
On 2014-05-07 06:33, Per Jessen wrote:
Carlos E. R. wrote:
How big is the email itself?
"Just" 19 MB. Not that /huge/.
[...]
SPAMD_AWL=no SPAMD_NICE=yes SPAMD_ARGS="-d -c --max-children=8 " SPAM_SA_UPDATE="no" SPAM_SA_COMPILE="no" SPAM_AMAVISD_RESTART="no" SPAM_SPAMD_RESTART="yes"
Carlos, I've not read this entire thread, but I thought that it might be useful if I mentioned that I have spamd running on a Raspberry Pi filtering ALL my incoming mail from 5 different accounts, and the load average rarely gets above 0.05. RAM usage never gets above 25% (of 512MB). The "magic bullet" on this was a) use spamc/spamd instead of calling spamassassin for every mail (which I note that you are doing) and b) use COMPILED rulesets (I see above that SPAM_SA_COMPILE is set to "no"). When I moved it to compiled rulesets the load average dropped from 28 (no, that is not a typo) to 0.05. I think the maximum of children it is allowed to spawn is 5. Try running sa-compile on your ruleset and set SPAM_SA_COMPILE to "yes" and see if that makes any difference for your setup. After about a week of experimenting when I first set it up on the Rpi I was just about ready to give it up as impossible; the compiled rulesets were the key. BTW, that Rpi runs fetchmail, sendmail (yes, sendmail 8.1.5; not postfix - I could never get along with it), procmail, spamassassin, clamav and dovecot and handles all my email (including remote access from outside). Ticks along beautifully, makes no noise and uses less than 5W of power. :) Regards, Rodney. -- ============================================================== Rodney Baker VK5ZTV rodney.baker@iinet.net.au ============================================================== -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/08/2014 09:20 AM, Rodney Baker wrote:
The "magic bullet" on this was a) use spamc/spamd instead of calling spamassassin for every mail (which I note that you are doing) and b) use COMPILED rulesets (I see above that SPAM_SA_COMPILE is set to "no").
When I moved it to compiled rulesets the load average dropped from 28 (no, that is not a typo) to 0.05. I think the maximum of children it is allowed to spawn is 5.
Hmm. Trying that now but it doesn't seem to offer any improvement. -- "Now look," Forrester said patiently, "progress is an outmoded idea. We've got to be in step with the times. We've got to ask ourselves what progress ever did for us." -- Randall Garett, "Pagan Passions", 1959 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (8)
-
Anton Aylward -
Carlos E. R. -
Carlos E. R. -
Cristian Rodríguez -
Linda Walsh -
Patrick Shanahan -
Per Jessen -
Rodney Baker