[opensuse] Big memory footprint: clamd
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, When I built my computer I put as much RAM as the board would allow, 8GiB. Today that is too little, but I can't add more. Telling 'mc' to sort by Resident memory, it shows: top - 22:44:32 up 28 days, 4:37, 4 users, load average: 0,26, 0,55, 1,02 Tasks: 521 total, 2 running, 517 sleeping, 0 stopped, 2 zombie %Cpu(s): 1,2 us, 0,6 sy, 0,0 ni, 97,3 id, 0,8 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem: 8174408 total, 7511580 used, 662828 free, 546440 buffers KiB Swap: 43744244 total, 1921984 used, 41822260 free. 3837656 cached Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 20581 cer 20 0 3553104 1,266g 263636 0 R 4,088 16,24 3:52.94 firefox 3493 vscan 20 0 1376180 536972 10364 14548 S 0,000 6,569 11:27.15 clamd 19773 cer 20 0 1452508 309508 141740 0 S 0,000 3,786 0:05.72 soffice.bin 3874 root 20 0 1090892 226564 154380 217968 S 0,798 2,772 182:22.08 X 31344 cer 20 0 206044 87728 7404 1664 S 0,000 1,073 1:55.04 alpine 4176 cer 20 0 748016 56796 21004 17136 S 0,000 0,695 2:56.67 xfdesktop 20088 cer 20 0 829220 53820 4596 0 S 0,000 0,658 0:02.04 gnome-software 9835 cer 20 0 784100 40604 17784 36520 S 0,100 0,497 1:46.40 gnote 4207 cer 20 0 891040 38988 22504 18136 S 0,000 0,477 0:32.57 keepassxc ... The worst offender this moment is Firefox (Thunderbird is closed now). But the next is clamd (clamav, antivirus mail scanner daemon)! It is using more than LibreOffice! How can that be? It is iddle at the moment, doing nothing. What can be done to reduce its footprint? Instead of the daemon, I could call it from amavis, one process per mail. But this impacts scan speed. Could the daemon be started "on demand", and be killed a minute after the last email goes through? - -- Cheers Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlmGMxIACgkQtTMYHG2NR9UeCgCeNA+LGMhfhjS84U4h7EeEwlbT RhkAoIldf2vOQVms3CwHLc87ybN7vVDs =KoWk -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2017-08-05 at 23:05 +0200, Carlos E. R. wrote: ...
The worst offender this moment is Firefox (Thunderbird is closed now). But the next is clamd (clamav, antivirus mail scanner daemon)! It is using more than LibreOffice!
How can that be? It is iddle at the moment, doing nothing.
What can be done to reduce its footprint?
Instead of the daemon, I could call it from amavis, one process per mail. But this impacts scan speed.
And amavis first tries the daemon, several times, before giving up and starting the process. Also, even if clamd service is not enabled, restarting amavis automatically starts clamd. Some sort of dependency.
Could the daemon be started "on demand", and be killed a minute after the last email goes through?
I googled, and it seems that yes, clamd uses a lot of memory for the databases, and it can not be swapped out. Data, not code. However, I still don't understand why it remains in RAM hours after the last email was processed. However, I see it gets called on socket "/var/run/clamav/clamd-socket". I wonder if it would be possible to use systemd to load the daemon when something tries that socket :-? Hw is that done? - -- Cheers, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlmH2qoACgkQtTMYHG2NR9VQ+ACfX58JzMMdKO6DlWNfJ3I4pxO4 0XsAni8aifApMk5beEEPLp7CVA365TdQ =mI8B -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
Could the daemon be started "on demand", and be killed a minute after the last email goes through?
I googled, and it seems that yes, clamd uses a lot of memory for the databases, and it can not be swapped out. Data, not code. However, I still don't understand why it remains in RAM hours after the last email was processed.
Possibly due to mlock(). Memory can be locked in. Or maybe clamd touches all of the database regularly. -- Per Jessen, Zürich (15.8°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-08-07 08:39, Per Jessen wrote:
Carlos E. R. wrote:
Could the daemon be started "on demand", and be killed a minute after the last email goes through?
I googled, and it seems that yes, clamd uses a lot of memory for the databases, and it can not be swapped out. Data, not code. However, I still don't understand why it remains in RAM hours after the last email was processed.
Possibly due to mlock(). Memory can be locked in. Or maybe clamd touches all of the database regularly.
My google finds (re clamd) say that you can not swap out a portion of the databases, because you do not know what will come in a file to scan. That I can understand, while scanning everything should be loaded in ram. But when nothing has been scanned for many minutes? People setting up mailserver with small machines had to enlarge ram just for clamd, or it would crash. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Carlos E. R. wrote:
On 2017-08-07 08:39, Per Jessen wrote:
Carlos E. R. wrote:
Could the daemon be started "on demand", and be killed a minute after the last email goes through?
I googled, and it seems that yes, clamd uses a lot of memory for the databases, and it can not be swapped out. Data, not code. However, I still don't understand why it remains in RAM hours after the last email was processed.
Possibly due to mlock(). Memory can be locked in. Or maybe clamd touches all of the database regularly.
My google finds (re clamd) say that you can not swap out a portion of the databases, because you do not know what will come in a file to scan. That I can understand, while scanning everything should be loaded in ram. But when nothing has been scanned for many minutes?
A quick grep through the libclamav source shows no mlock() calls. Maybe clamav wanders through the memory every so often? (like my code from my other posting) You could check to see what clamd does with an strace.
People setting up mailserver with small machines had to enlarge ram just for clamd, or it would crash.
768Mb per mailserver is what I used until a few months ago. Since 2006. It was a little too tight. -- Per Jessen, Zürich (21.2°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-08-07 14:27, Per Jessen wrote:
Carlos E. R. wrote:
On 2017-08-07 08:39, Per Jessen wrote:
Carlos E. R. wrote:
Could the daemon be started "on demand", and be killed a minute after the last email goes through?
I googled, and it seems that yes, clamd uses a lot of memory for the databases, and it can not be swapped out. Data, not code. However, I still don't understand why it remains in RAM hours after the last email was processed.
Possibly due to mlock(). Memory can be locked in. Or maybe clamd touches all of the database regularly.
My google finds (re clamd) say that you can not swap out a portion of the databases, because you do not know what will come in a file to scan. That I can understand, while scanning everything should be loaded in ram. But when nothing has been scanned for many minutes?
A quick grep through the libclamav source shows no mlock() calls. Maybe clamav wanders through the memory every so often? (like my code from my other posting)
You could check to see what clamd does with an strace.
Maybe, for scientific interest. You can say that 500 MB is not that large a memory. Well, it is 1/16 of all the memory I have and it is frozen used all the time... I still have to look at systemd examples, I just need to find the right state of mind to do so ;-) -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On 07/08/17 06:12, Carlos E. R. wrote:
clamd uses a lot of memory for the databases
- during this hot weather , running Clam AV causes 4-core CPU to run extremely hot , often causing 100 C shutdowns regards -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
ellanios82 wrote:
On 07/08/17 06:12, Carlos E. R. wrote:
clamd uses a lot of memory for the databases
- during this hot weather , running Clam AV causes 4-core CPU to run extremely hot , often causing 100 C shutdowns
Poor airflow? I have four clamd instances running in each their host on Xen, 4 cpus, 8Gb memory - roomtemp approx 25C. clamd is cpu intensive, but unless it's got something to do (constant flow of emails for instance), it shouldn't be causing any excessive cpu usage. -- Per Jessen, Zürich (16.9°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 07/08/17 10:55, Per Jessen wrote:
ellanios82 wrote:
On 07/08/17 06:12, Carlos E. R. wrote:
clamd uses a lot of memory for the databases
- during this hot weather , running Clam AV causes 4-core CPU to run extremely hot , often causing 100 C shutdowns Poor airflow? I have four clamd instances running in each their host on Xen, 4 cpus, 8Gb memory - roomtemp approx 25C. clamd is cpu intensive, but unless it's got something to do (constant flow of emails for instance), it shouldn't be causing any excessive cpu usage.
- aha - tks : : uncomfortably, roomtemp here 35C [ and, for next 7 days , noon temps 40C monotonously expected.] .... regards -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-08-07 09:40, ellanios82 wrote:
On 07/08/17 06:12, Carlos E. R. wrote:
clamd uses a lot of memory for the databases
- during this hot weather , running Clam AV causes 4-core CPU to run extremely hot , often causing 100 C shutdowns
That should never happen. First, the machine should have enough radiator surface and airflow to avoid overheating even at full cpu for hours. Secondly, in case of over temperature, the cpu should slow down the clock sufficiently and smartly. Slow down the clock, even insert dummy "noops" instructions, to give the cpu time to shed heat without having to shut down. This the CPU can do by itself, or the kernel can do it. My small mini-pc which I use for multimedia and server does this trick. I noticed when watching some movies that the player would skip frames when hot: the clock had slowed down and the player could then not cope. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Carlos E. R. wrote:
The worst offender this moment is Firefox (Thunderbird is closed now). But the next is clamd (clamav, antivirus mail scanner daemon)! It is using more than LibreOffice!
How can that be? It is iddle at the moment, doing nothing.
The signature databases take up the space. I have a few instances running, all having about 500M resident. (your's takes up a lot more in total, but I'm running a custom-written daemon).
What can be done to reduce its footprint?
Cut down on the databases. Are you using any 3rd party databases (sanesecurity for instance) ?
Instead of the daemon, I could call it from amavis, one process per mail. But this impacts scan speed. Could the daemon be started "on demand", and be killed a minute after the last email goes through?
Perhaps, but you're looking to save 500Mb on a machine with 8Gb? Looking at your 'top' display, where does the rest of those 7Gb go? -- Per Jessen, Zürich (15.4°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-08-07 08:33, Per Jessen wrote:
Carlos E. R. wrote:
The worst offender this moment is Firefox (Thunderbird is closed now). But the next is clamd (clamav, antivirus mail scanner daemon)! It is using more than LibreOffice!
How can that be? It is iddle at the moment, doing nothing.
The signature databases take up the space. I have a few instances running, all having about 500M resident. (your's takes up a lot more in total, but I'm running a custom-written daemon).
What can be done to reduce its footprint?
Cut down on the databases. Are you using any 3rd party databases (sanesecurity for instance) ?
I'm using defaults, I think. Yes, the config file is dated dec 27 2016, so it is the installation file. I do not find any "sane" string in the config.
Instead of the daemon, I could call it from amavis, one process per mail. But this impacts scan speed. Could the daemon be started "on demand", and be killed a minute after the last email goes through?
Perhaps, but you're looking to save 500Mb on a machine with 8Gb?
I notice those 500, believe me. The machine responds a bit better just after I killed clamd.
Looking at your 'top' display, where does the rest of those 7Gb go?
Many things, it is just that clamav is the biggest chunk except for firefox and thunderbird. KiB Mem: 8174408 total, 4992444 used, 3181964 free, 103452 buffers KiB Swap: 43744244 total, 3190824 used, 40553420 free. 1909944 cached Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 20581 cer 20 0 4573608 1,064g 96912 416792 S 2,970 13,65 164:33.60 firefox 9866 cer 20 0 3045056 876360 73076 61684 S 0,000 10,72 25:26.60 thunderbird-bin 19773 cer 20 0 6670680 437464 41680 166220 S 0,330 5,352 10:14.16 soffice.bin 3874 root 20 0 1219596 161992 94228 234648 S 1,650 1,982 198:42.50 X 13111 cer 20 0 5270092 52544 9988 56176 S 0,330 0,643 0:22.19 java 4176 cer 20 0 754416 51876 18528 25848 S 0,000 0,635 3:11.76 xfdesktop 31344 cer 20 0 206044 33584 4936 53340 S 0,000 0,411 2:26.57 alpine 4210 cer 20 0 607528 29500 9720 39220 S 1,320 0,361 48:43.08 xfce4-terminal 4345 cer 20 0 509716 28680 16732 17996 S 0,000 0,351 0:04.38 Thunar 32676 vscan 20 0 166836 27992 5540 33220 S 0,000 0,342 0:00.30 /usr/sbin/amavi 17649 cer 20 0 424908 27712 6244 17432 S 0,000 0,339 0:28.00 kwalletd 4149 cer 20 0 510288 23400 9204 34344 S 0,000 0,286 9:15.91 xfce4-panel 3370 named 20 0 436824 19368 252 43828 S 0,000 0,237 3:41.93 named 4352 cer 39 19 1080232 18360 3108 13396 S 0,000 0,225 3:38.41 tracker-miner-f After that, the "RES" column has less than 20000 (clamd is manually stopped). Clamd has a bigger mem print than soffice.bin Notice that I have 3 G in swap. I would also benefit by using a very fast swap disk. That's another thing: I noticed that Leap reads swap very very slow. It can take two or four minutes for big apps (firefox, Libreoffice) to start responding after hibernation. 13.1 was much faster in this, almost instant response. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Hi, Am 07.08.2017 um 12:42 schrieb Carlos E. R.:
Many things, it is just that clamav is the biggest chunk except for firefox and thunderbird.
just to throw that in. Antivirus/Antispam is a quite expensive (as in memory, cpu and maintenance) function. As a mailserver admin this is something which is somehow expected. If you run clamav on an extremely low traffic mailserver there still should be the option to run it on demand e.g. from amavisd-new. Have you looked into that option? Wolfgang -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-08-07 12:56, Wolfgang Rosenauer wrote:
Hi,
Am 07.08.2017 um 12:42 schrieb Carlos E. R.:
Many things, it is just that clamav is the biggest chunk except for firefox and thunderbird.
just to throw that in. Antivirus/Antispam is a quite expensive (as in memory, cpu and maintenance) function. As a mailserver admin this is something which is somehow expected.
Yes, that I understand.
If you run clamav on an extremely low traffic mailserver there still should be the option to run it on demand e.g. from amavisd-new. Have you looked into that option?
clamd is installed in the same clamav rpm, so it can't be removed. If clamd is not running, amavis tries insistently to reach it a few times before giving up and calling clamav or clamscan directly as a separate process. This is repeated for each mail, I think. And of course, starting clamav once per mail is heavy on a fetchmail run, which can pick a thousand emails in few minutes. I do not see how to configure amavis for not using clamd. If I start amavis, it starts clamd as a dependency. The best idea seems to be to configure systemd to start clamd by socket and stop it by timer. I need a howto for dummies on this. :-) -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Am 07.08.2017 um 13:05 schrieb Carlos E. R.:
If you run clamav on an extremely low traffic mailserver there still should be the option to run it on demand e.g. from amavisd-new. Have you looked into that option?
clamd is installed in the same clamav rpm, so it can't be removed. If clamd is not running, amavis tries insistently to reach it a few times before giving up and calling clamav or clamscan directly as a separate process. This is repeated for each mail, I think. And of course, starting clamav once per mail is heavy on a fetchmail run, which can pick a thousand emails in few minutes.
I do not see how to configure amavis for not using clamd. If I start amavis, it starts clamd as a dependency.
Really? It used to be the way that you were able to comment the ClamAV-clamd entry in /etc/amavisd.conf and instead activate the clamscan in the same file. On the other hand I see a Wants=named.service clamd.service which does not allow that setup :-( I think this hard requirement of Wants=clamd.service is actually too much in any case. I tend to remove clamd.service from Wants unless someone has a smarter idea. Carlos, you can try locally as well if you like.
The best idea seems to be to configure systemd to start clamd by socket and stop it by timer. I need a howto for dummies on this. :-)
I'm not a systemd expert on it. clamav is not my package but amavisd-new is somehow. An optional socket activation actually sounds useful in any case. Please ask the maintainer or via bugzilla explicitely what he thinks about it. Wolfgang -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-08-07 13:24, Wolfgang Rosenauer wrote:
Am 07.08.2017 um 13:05 schrieb Carlos E. R.:
If you run clamav on an extremely low traffic mailserver there still should be the option to run it on demand e.g. from amavisd-new. Have you looked into that option?
clamd is installed in the same clamav rpm, so it can't be removed. If clamd is not running, amavis tries insistently to reach it a few times before giving up and calling clamav or clamscan directly as a separate process. This is repeated for each mail, I think. And of course, starting clamav once per mail is heavy on a fetchmail run, which can pick a thousand emails in few minutes.
I do not see how to configure amavis for not using clamd. If I start amavis, it starts clamd as a dependency.
Really? It used to be the way that you were able to comment the ClamAV-clamd entry in /etc/amavisd.conf and instead activate the clamscan in the same file.
I tried... either I did it wrong, or did not work. # ### http://www.clamav.net/ #['ClamAV-clamd', # \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd-socket"], # qr/\bOK$/m, qr/\bFOUND$/m, # qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ], I commented out that section.
On the other hand I see a Wants=named.service clamd.service which does not allow that setup :-(
Yes, that's what I supposed.
I think this hard requirement of Wants=clamd.service is actually too much in any case.
I tend to remove clamd.service from Wants unless someone has a smarter idea. Carlos, you can try locally as well if you like.
It is a possibility, yes.
The best idea seems to be to configure systemd to start clamd by socket and stop it by timer. I need a howto for dummies on this. :-)
I'm not a systemd expert on it. clamav is not my package but amavisd-new is somehow. An optional socket activation actually sounds useful in any case. Please ask the maintainer or via bugzilla explicitely what he thinks about it.
My idea was to find out how to do it, here, experiment, then suggest the optional improvement to the maintainer. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Carlos E. R. wrote:
I think this hard requirement of Wants=clamd.service is actually too much in any case.
I tend to remove clamd.service from Wants unless someone has a smarter idea. Carlos, you can try locally as well if you like.
It is a possibility, yes.
Seems to be the easiest way out - provided you can get amavis to auto-start clamd: in /etc/systemd/system/clamd.service.d/carlos.conf: [Service] Wants=
My idea was to find out how to do it, here, experiment, then suggest the optional improvement to the maintainer.
A pretty long and comprehensive page: https://www.freedesktop.org/software/systemd/man/systemd.socket.html Maybe there already examples of socket activated apps out there? -- Per Jessen, Zürich (22.4°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-08-07 14:44, Per Jessen wrote:
Carlos E. R. wrote:
I think this hard requirement of Wants=clamd.service is actually too much in any case.
I tend to remove clamd.service from Wants unless someone has a smarter idea. Carlos, you can try locally as well if you like.
It is a possibility, yes.
Seems to be the easiest way out - provided you can get amavis to auto-start clamd:
in /etc/systemd/system/clamd.service.d/carlos.conf:
[Service] Wants=
My idea was to find out how to do it, here, experiment, then suggest the optional improvement to the maintainer.
A pretty long and comprehensive page: https://www.freedesktop.org/software/systemd/man/systemd.socket.html
Thanks, I'll read that.
Maybe there already examples of socket activated apps out there?
I guess so... -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On 2017-08-07 14:52, Carlos E. R. wrote:
On 2017-08-07 14:44, Per Jessen wrote:
Carlos E. R. wrote:
I think this hard requirement of Wants=clamd.service is actually too much in any case.
I tend to remove clamd.service from Wants unless someone has a smarter idea. Carlos, you can try locally as well if you like.
It is a possibility, yes.
Seems to be the easiest way out - provided you can get amavis to auto-start clamd:
in /etc/systemd/system/clamd.service.d/carlos.conf:
[Service] Wants=
Ah, I forgot about that method, thanks. I was thinking of editing the master file :-(
My idea was to find out how to do it, here, experiment, then suggest the optional improvement to the maintainer.
A pretty long and comprehensive page: https://www.freedesktop.org/software/systemd/man/systemd.socket.html
Thanks, I'll read that.
Oh. It is a man page :-( List of options, no examples.
Maybe there already examples of socket activated apps out there?
I guess so...
I'll have to find them. Time for lunch and other things, like getting some nespresso capsules. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Carlos E. R. wrote:
On 2017-08-07 14:52, Carlos E. R. wrote:
My idea was to find out how to do it, here, experiment, then suggest the optional improvement to the maintainer.
A pretty long and comprehensive page: https://www.freedesktop.org/software/systemd/man/systemd.socket.html
Thanks, I'll read that.
Oh. It is a man page :-( List of options, no examples.
Yeah. The reference. On Leap422, I looked for examples - maybe look at avahi-daemon, that looks good. Not sure about the automatic shutdown though. -- Per Jessen, Zürich (23.3°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-08-07 12:42, Carlos E. R. wrote:
On 2017-08-07 08:33, Per Jessen wrote:
Instead of the daemon, I could call it from amavis, one process per mail. But this impacts scan speed. Could the daemon be started "on demand", and be killed a minute after the last email goes through?
Perhaps, but you're looking to save 500Mb on a machine with 8Gb?
I notice those 500, believe me. The machine responds a bit better just after I killed clamd.
I see two avenues. The first one is to modify my "get mail" script to first start clamd service, and then manually stop clamd service after mail has been processed (I do not yet easily see how to stop it automatically). But clamd gets also called when a mail goes out (unneeded, but it does so). The other method, much cleaner, would be to start service when the socket gets something, which I believe systemd can do but I have never done; no idea where to start. Guide for dummies? Then stop it after a minute of no use, also automatically. Is this possible? -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Carlos E. R. wrote:
On 2017-08-07 08:33, Per Jessen wrote:
Cut down on the databases. Are you using any 3rd party databases (sanesecurity for instance) ?
I'm using defaults, I think. Yes, the config file is dated dec 27 2016, so it is the installation file. I do not find any "sane" string in the config.
It wouldn't be in the config, clamd will load everything in the database dir. If you haven't added sanesecurity yourself, they're not there. You would also need a regular refresh etc. So it's probably just the default install - I don't have any historical data, but I have had clamd running on physical machines with 768Mb for years, now recently virtualised with about 1Gb each.
Instead of the daemon, I could call it from amavis, one process per mail. But this impacts scan speed. Could the daemon be started "on demand", and be killed a minute after the last email goes through?
Perhaps, but you're looking to save 500Mb on a machine with 8Gb?
I notice those 500, believe me. The machine responds a bit better just after I killed clamd.
Perhaps, but there must other processes gobbling up the rest, that's what I meant. The 500Mb for clamd is normal.
Looking at your 'top' display, where does the rest of those 7Gb go?
Many things, it is just that clamav is the biggest chunk except for firefox and thunderbird.
KiB Mem: 8174408 total, 4992444 used, 3181964 free, 103452 buffers KiB Swap: 43744244 total, 3190824 used, 40553420 free. 1909944 cached Mem
8Gb physical, 5Gb used, 3Gb swap and 2Gb file system buffers. Firefox and Thunderbird are also the biggest consumers on my main desktop, following by Xorg, konqueror and soffice. The rest less than 50Mb resident per process. Total: Mem: 4054784k total, 2976472k used, 1078312k free, 16k buffers Swap: 3911736k total, 778436k used, 3133300k free, 1113848k cached
Notice that I have 3 G in swap. I would also benefit by using a very fast swap disk.
Just add more memory, but you said you can't. Unless we're talking video editing and gaming (or something else that gobbles of memory), 4Gb should be more than plenty. Well, it is on my main desktop - not running Leap though. -- Per Jessen, Zürich (20.4°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-08-07 13:20, Per Jessen wrote:
Carlos E. R. wrote:
On 2017-08-07 08:33, Per Jessen wrote:
Cut down on the databases. Are you using any 3rd party databases (sanesecurity for instance) ?
I'm using defaults, I think. Yes, the config file is dated dec 27 2016, so it is the installation file. I do not find any "sane" string in the config.
It wouldn't be in the config, clamd will load everything in the database dir. If you haven't added sanesecurity yourself, they're not there.
No such package here.
You would also need a regular refresh etc. So it's probably just the default install - I don't have any historical data, but I have had clamd running on physical machines with 768Mb for years, now recently virtualised with about 1Gb each.
Instead of the daemon, I could call it from amavis, one process per mail. But this impacts scan speed. Could the daemon be started "on demand", and be killed a minute after the last email goes through?
Perhaps, but you're looking to save 500Mb on a machine with 8Gb?
I notice those 500, believe me. The machine responds a bit better just after I killed clamd.
Perhaps, but there must other processes gobbling up the rest, that's what I meant. The 500Mb for clamd is normal.
I know that it is normal, for clamd. Still, it is a lot, it is the third biggest process in my machine. Doing nothing.
Looking at your 'top' display, where does the rest of those 7Gb go?
Many things, it is just that clamav is the biggest chunk except for firefox and thunderbird.
KiB Mem: 8174408 total, 4992444 used, 3181964 free, 103452 buffers KiB Swap: 43744244 total, 3190824 used, 40553420 free. 1909944 cached Mem
8Gb physical, 5Gb used, 3Gb swap and 2Gb file system buffers.
And 3 free.
Firefox and Thunderbird are also the biggest consumers on my main desktop, following by Xorg, konqueror and soffice. The rest less than 50Mb resident per process. Total:
Mem: 4054784k total, 2976472k used, 1078312k free, 16k buffers Swap: 3911736k total, 778436k used, 3133300k free, 1113848k cached
Notice that I have 3 G in swap. I would also benefit by using a very fast swap disk.
Just add more memory, but you said you can't. Unless we're talking video editing and gaming (or something else that gobbles of memory), 4Gb should be more than plenty. Well, it is on my main desktop - not running Leap though.
Leap seems to be very slow when swapping. The same machine with 13.1 was noticeably faster. I like to have all the applications I use open in a separate workspace, with some unfinished task, so I go back to them later. Thus, firefox, thunderbid and LO cn be opened at the same time. There is a java process that controls my TV. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Carlos E. R. wrote:
On 2017-08-07 13:20, Per Jessen wrote:
Carlos E. R. wrote:
On 2017-08-07 08:33, Per Jessen wrote:
Cut down on the databases. Are you using any 3rd party databases (sanesecurity for instance) ?
I'm using defaults, I think. Yes, the config file is dated dec 27 2016, so it is the installation file. I do not find any "sane" string in the config.
It wouldn't be in the config, clamd will load everything in the database dir. If you haven't added sanesecurity yourself, they're not there.
No such package here.
it's not packaged, sanesecurity maintains 3rd party signatures for clamav: http://sanesecurity.com/ I expect there are other databases out there too.
Perhaps, but there must other processes gobbling up the rest, that's what I meant. The 500Mb for clamd is normal.
I know that it is normal, for clamd. Still, it is a lot, it is the third biggest process in my machine. Doing nothing.
Does it do any regular scanning of your files? Just thinking out loud.
Looking at your 'top' display, where does the rest of those 7Gb go?
Many things, it is just that clamav is the biggest chunk except for firefox and thunderbird.
KiB Mem: 8174408 total, 4992444 used, 3181964 free, 103452 buffers KiB Swap: 43744244 total, 3190824 used, 40553420 free. 1909944 cached Mem
8Gb physical, 5Gb used, 3Gb swap and 2Gb file system buffers.
And 3 free.
Right, 8 - 5 = 3. Under those conditions, it's difficult to imagine 500Mb more or less free memory should make any noticable difference. You can try it yourself - int main( int argc, char **argv ) { int sz, p; char *f; sz=512*1024*1024L; f=malloc(sz); while( 1 ) { for( p=0; p<sz; p+=4096 ) f[p]=0; sleep(30); } return 0; } Running it, then killing it and freeing the 512Mb should not be noticable unless your system is swapping.
Just add more memory, but you said you can't. Unless we're talking video editing and gaming (or something else that gobbles of memory), 4Gb should be more than plenty. Well, it is on my main desktop - not running Leap though.
Leap seems to be very slow when swapping. The same machine with 13.1 was noticeably faster.
You said that before, but in relation to resuming from hibernation. That's not quite the same. Besides, your machine is not swapping?
I like to have all the applications I use open in a separate workspace, with some unfinished task, so I go back to them later.
Me too.
Thus, firefox, thunderbid and LO cn be opened at the same time. There is a java process that controls my TV.
Same here, except for the java. -- Per Jessen, Zürich (21.2°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-08-07 14:24, Per Jessen wrote:
Carlos E. R. wrote:
On 2017-08-07 13:20, Per Jessen wrote:
Carlos E. R. wrote:
On 2017-08-07 08:33, Per Jessen wrote:
Cut down on the databases. Are you using any 3rd party databases (sanesecurity for instance) ?
I'm using defaults, I think. Yes, the config file is dated dec 27 2016, so it is the installation file. I do not find any "sane" string in the config.
It wouldn't be in the config, clamd will load everything in the database dir. If you haven't added sanesecurity yourself, they're not there.
No such package here.
it's not packaged, sanesecurity maintains 3rd party signatures for clamav:
I expect there are other databases out there too.
I don't recall activating such a thing, no.
Perhaps, but there must other processes gobbling up the rest, that's what I meant. The 500Mb for clamd is normal.
I know that it is normal, for clamd. Still, it is a lot, it is the third biggest process in my machine. Doing nothing.
Does it do any regular scanning of your files? Just thinking out loud.
No, I don't do that, no need. There is no smb serving of windows machine that would need such a thing.
Looking at your 'top' display, where does the rest of those 7Gb go?
Many things, it is just that clamav is the biggest chunk except for firefox and thunderbird.
KiB Mem: 8174408 total, 4992444 used, 3181964 free, 103452 buffers KiB Swap: 43744244 total, 3190824 used, 40553420 free. 1909944 cached Mem
8Gb physical, 5Gb used, 3Gb swap and 2Gb file system buffers.
And 3 free.
Right, 8 - 5 = 3. Under those conditions, it's difficult to imagine 500Mb more or less free memory should make any noticable difference. You can try it yourself -
int main( int argc, char **argv ) { int sz, p; char *f;
sz=512*1024*1024L; f=malloc(sz);
while( 1 ) { for( p=0; p<sz; p+=4096 ) f[p]=0; sleep(30); }
return 0; }
Running it, then killing it and freeing the 512Mb should not be noticable unless your system is swapping.
But I did notice when I stopped clamd that libreoffice responded faster. It depends what the load at the moment was, of course. This moment it is: KiB Mem: 8174408 total, 5608116 used, 2566292 free, 207024 buffers KiB Swap: 43744244 total, 3097036 used, 40647208 free. 2419792 cached Mem was: KiB Mem: 8174408 total, 4992444 used, 3181964 free, 103452 buffers KiB Swap: 43744244 total, 3190824 used, 40553420 free. 1909944 cached Mem
Just add more memory, but you said you can't. Unless we're talking video editing and gaming (or something else that gobbles of memory), 4Gb should be more than plenty. Well, it is on my main desktop - not running Leap though.
Leap seems to be very slow when swapping. The same machine with 13.1 was noticeably faster.
You said that before, but in relation to resuming from hibernation. That's not quite the same. Besides, your machine is not swapping?
Yes, 3 GB in swap. See it has gone up during the morning. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Carlos E. R. wrote: [snip]
KiB Mem: 8174408 total, 5608116 used, 2566292 free, 207024 buffers KiB Swap: 43744244 total, 3097036 used, 40647208 free. 2419792 cached Mem
was:
KiB Mem: 8174408 total, 4992444 used, 3181964 free, 103452 buffers KiB Swap: 43744244 total, 3190824 used, 40553420 free. 1909944 cached Mem
Just add more memory, but you said you can't. Unless we're talking video editing and gaming (or something else that gobbles of memory), 4Gb should be more than plenty. Well, it is on my main desktop - not running Leap though.
Leap seems to be very slow when swapping. The same machine with 13.1 was noticeably faster.
You said that before, but in relation to resuming from hibernation. That's not quite the same. Besides, your machine is not swapping?
Yes, 3 GB in swap. See it has gone up during the morning.
"swapping" is more the situation where there is no or very little free memory. What you're seeing is more "has swapped in the past". If it were my system, I would look at that first. An 8Gb system with some 3Gb swap in use. Unless you know that you're running it to the max. None of our desktops have more than 4Gb memory, but they're not running Leap either. I've just had to re-allocate my main test system, I'll have to get another started. -- Per Jessen, Zürich (23.2°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Remember this, when we wrote about the large memory footprint of clamd? Well, it no longer happens, clamd is swapped out soon after a mail fetch run: top - 19:33:22 up 11 days, 7:47, 1 user, load average: 0,44, 0,58, 0,93 Tasks: 520 total, 1 running, 517 sleeping, 0 stopped, 2 zombie %Cpu(s): 1,6 us, 0,6 sy, 0,0 ni, 96,6 id, 1,2 wa, 0,0 hi, 0,1 si, 0,0 st KiB Mem: 8174404 total, 2576192 used, 5598212 free, 45080 buffers KiB Swap: 25165820 total, 3958492 used, 21207328 free. 1152372 cached Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 3884 vscan 20 0 1054360 15528 0 519464 S 0,000 0,190 6:17.80 clamd 7269 cer 20 0 4083836 490408 53260 334432 S 0,331 5,999 250:57.99 thunderbird-bin (sorted by swap usage) I did nothing in the end. Perhaps some update altered the way clamd behaves regarding memory :-? If it was some update, thanks :-) On Saturday, 2017-08-05 at 23:05 +0200, Carlos E. R. wrote:
Hi,
When I built my computer I put as much RAM as the board would allow, 8GiB. Today that is too little, but I can't add more.
Telling 'mc' to sort by Resident memory, it shows:
(top)
top - 22:44:32 up 28 days, 4:37, 4 users, load average: 0,26, 0,55, 1,02 Tasks: 521 total, 2 running, 517 sleeping, 0 stopped, 2 zombie %Cpu(s): 1,2 us, 0,6 sy, 0,0 ni, 97,3 id, 0,8 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem: 8174408 total, 7511580 used, 662828 free, 546440 buffers KiB Swap: 43744244 total, 1921984 used, 41822260 free. 3837656 cached Mem
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 20581 cer 20 0 3553104 1,266g 263636 0 R 4,088 16,24 3:52.94 firefox 3493 vscan 20 0 1376180 536972 10364 14548 S 0,000 6,569 11:27.15 clamd
...
The worst offender this moment is Firefox (Thunderbird is closed now). But the next is clamd (clamav, antivirus mail scanner daemon)! It is using more than LibreOffice!
How can that be? It is iddle at the moment, doing nothing.
What can be done to reduce its footprint?
Instead of the daemon, I could call it from amavis, one process per mail. But this impacts scan speed.
Could the daemon be started "on demand", and be killed a minute after the last email goes through?
- -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlo1aHMACgkQtTMYHG2NR9X0vgCfXW5mkoKuQ7YJbm5Loapej43K NWEAnA0QvbYr5ErvcUer+fK33NVcSz5d =WxYY -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
Remember this, when we wrote about the large memory footprint of clamd?
Well, it no longer happens, clamd is swapped out soon after a mail fetch run:
top - 19:33:22 up 11 days, 7:47, 1 user, load average: 0,44, 0,58, 0,93 Tasks: 520 total, 1 running, 517 sleeping, 0 stopped, 2 zombie %Cpu(s): 1,6 us, 0,6 sy, 0,0 ni, 96,6 id, 1,2 wa, 0,0 hi, 0,1 si, 0,0 st KiB Mem: 8174404 total, 2576192 used, 5598212 free, 45080 buffers KiB Swap: 25165820 total, 3958492 used, 21207328 free. 1152372 cached Mem
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 3884 vscan 20 0 1054360 15528 0 519464 S 0,000 0,190 6:17.80 clamd 7269 cer 20 0 4083836 490408 53260 334432 S 0,331 5,999 250:57.99 thunderbird-bin
(sorted by swap usage)
I did nothing in the end. Perhaps some update altered the way clamd behaves regarding memory :-?
Unlikely I would say. Besides, it isn't really the job for an application to determine when to be swapped and when to be out :-) Your situation does look weird - about 5Gb of free memory, so although clam isn't in use at the moment, there's no reason for it to be swapped out. -- Per Jessen, Zürich (1.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sunday, 2017-12-17 at 10:02 +0100, Per Jessen wrote:
Carlos E. R. wrote:
Remember this, when we wrote about the large memory footprint of clamd?
Well, it no longer happens, clamd is swapped out soon after a mail fetch run:
...
I did nothing in the end. Perhaps some update altered the way clamd behaves regarding memory :-?
Unlikely I would say. Besides, it isn't really the job for an application to determine when to be swapped and when to be out :-)
Your situation does look weird - about 5Gb of free memory, so although clam isn't in use at the moment, there's no reason for it to be swapped out.
Highly unlikely, true. That's why I asked for more eyes ;-) There was 5 GB free because I had closed firefox. It seems to me that an application may mark a memory not swappable, which clamd did. Guessing, it is possible that they now don't do that. But thinking after sending the post, it might be that I had intentionally disabled antivirus scan in amavis-new and I have forgotten. I'll try to verify now, fresh tea in hand. The file to look at would be /etc/amavisd.conf. It is dated Aug 11; this thread was in August, so it matches. And yes, the section seems commented out, so I'll undo. Then clamd can indeed be swapped out, just that it hardly does. Well, then the idea about starting the service on demand gets back to the table. - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlo2UcAACgkQtTMYHG2NR9VM8gCeKwUBWYxnUV3pgd+RwSV9iW3Z sPkAnA00Y5OqURcWmwUUiXJvn//GTwWd =SypX -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sunday, 2017-12-17 at 12:15 +0100, Carlos E. R. wrote:
On Sunday, 2017-12-17 at 10:02 +0100, Per Jessen wrote:
...
I'll try to verify now, fresh tea in hand.
The file to look at would be /etc/amavisd.conf. It is dated Aug 11; this thread was in August, so it matches. And yes, the section seems commented out, so I'll undo.
No, it is not that. I just compared with file from rpm and the section was untouched. Still, clamd seems to be not working, not scanning mail (judging from 'top' cpu view). - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlo2V6kACgkQtTMYHG2NR9VxHQCeM/+r2lCffZDI8vf/RfzQ/yqr ovwAniU0SsM2DgeWyErzIw4CEbCyTVAL =PfoU -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On Sunday, 2017-12-17 at 10:02 +0100, Per Jessen wrote:
Carlos E. R. wrote:
Remember this, when we wrote about the large memory footprint of clamd?
Well, it no longer happens, clamd is swapped out soon after a mail fetch run:
...
I did nothing in the end. Perhaps some update altered the way clamd behaves regarding memory :-?
Unlikely I would say. Besides, it isn't really the job for an application to determine when to be swapped and when to be out :-)
Your situation does look weird - about 5Gb of free memory, so although clam isn't in use at the moment, there's no reason for it to be swapped out.
Highly unlikely, true. That's why I asked for more eyes ;-) There was 5 GB free because I had closed firefox. It seems to me that an application may mark a memory not swappable, which clamd did. Guessing, it is possible that they now don't do that.
Even an unprivileged process can use mlock() I believe, but there is some limit to how much it can lock. Anyway, libclamav doesn't use mlock at all, hasn't done for 7 years or more. (I've just grepped the sources).
But thinking after sending the post, it might be that I had intentionally disabled antivirus scan in amavis-new and I have forgotten. The file to look at would be /etc/amavisd.conf. It is dated Aug 11; this thread was in August, so it matches. And yes, the section seems commented out, so I'll undo.
Aha. So it wouldn't be used at all?
Then clamd can indeed be swapped out, just that it hardly does.
Well, then the idea about starting the service on demand gets back to the table.
When you don't care much about performance, sure. (loading the clamd tables takes a few seconds). It still seems to me clamd will be swapped out when it's not in use and the memory is needed elsewhere, so your on-demand idea is essentially already in place? You could also try issuing a SIGSTOP clamd, that should eventually get it swapped out. -- Per Jessen, Zürich (1.8°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (4)
-
Carlos E. R. -
ellanios82 -
Per Jessen -
Wolfgang Rosenauer