On 2017-08-07 08:39, Per Jessen wrote:

Carlos E. R. wrote:

...

...

Could the daemon be started "on demand", and be killed a minute after the last email goes through?

I googled, and it seems that yes, clamd uses a lot of memory for the databases, and it can not be swapped out. Data, not code. However, I still don't understand why it remains in RAM hours after the last email was processed.

Possibly due to mlock(). Memory can be locked in. Or maybe clamd touches all of the database regularly.

My google finds (re clamd) say that you can not swap out a portion of the databases, because you do not know what will come in a file to scan. That I can understand, while scanning everything should be loaded in ram. But when nothing has been scanned for many minutes? People setting up mailserver with small machines had to enlarge ram just for clamd, or it would crash. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

On 2017-08-07 14:27, Per Jessen wrote:

Carlos E. R. wrote:

...

On 2017-08-07 08:39, Per Jessen wrote:

...

Carlos E. R. wrote:

...

...

Could the daemon be started "on demand", and be killed a minute after the last email goes through?

I googled, and it seems that yes, clamd uses a lot of memory for the databases, and it can not be swapped out. Data, not code. However, I still don't understand why it remains in RAM hours after the last email was processed.

Possibly due to mlock(). Memory can be locked in. Or maybe clamd touches all of the database regularly.

My google finds (re clamd) say that you can not swap out a portion of the databases, because you do not know what will come in a file to scan. That I can understand, while scanning everything should be loaded in ram. But when nothing has been scanned for many minutes?

A quick grep through the libclamav source shows no mlock() calls. Maybe clamav wanders through the memory every so often? (like my code from my other posting)

You could check to see what clamd does with an strace.

Maybe, for scientific interest. You can say that 500 MB is not that large a memory. Well, it is 1/16 of all the memory I have and it is frozen used all the time... I still have to look at systemd examples, I just need to find the right state of mind to do so ;-) -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

ellanios82 wrote:

On 07/08/17 06:12, Carlos E. R. wrote:

...

clamd uses a lot of memory for the databases

- during this hot weather , running Clam AV causes 4-core CPU to run extremely hot , often causing 100 C shutdowns

Poor airflow? I have four clamd instances running in each their host on Xen, 4 cpus, 8Gb memory - roomtemp approx 25C. clamd is cpu intensive, but unless it's got something to do (constant flow of emails for instance), it shouldn't be causing any excessive cpu usage. -- Per Jessen, Zürich (16.9°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2017-08-07 09:40, ellanios82 wrote:

On 07/08/17 06:12, Carlos E. R. wrote:

...

clamd uses a lot of memory for the databases

- during this hot weather , running Clam AV causes 4-core CPU to run extremely hot , often causing 100 C shutdowns

That should never happen. First, the machine should have enough radiator surface and airflow to avoid overheating even at full cpu for hours. Secondly, in case of over temperature, the cpu should slow down the clock sufficiently and smartly. Slow down the clock, even insert dummy "noops" instructions, to give the cpu time to shed heat without having to shut down. This the CPU can do by itself, or the kernel can do it. My small mini-pc which I use for multimedia and server does this trick. I noticed when watching some movies that the player would skip frames when hot: the clock had slowed down and the player could then not cope. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

On 2017-08-07 08:33, Per Jessen wrote:

Carlos E. R. wrote:

...

The worst offender this moment is Firefox (Thunderbird is closed now). But the next is clamd (clamav, antivirus mail scanner daemon)! It is using more than LibreOffice!

How can that be? It is iddle at the moment, doing nothing.

The signature databases take up the space. I have a few instances running, all having about 500M resident. (your's takes up a lot more in total, but I'm running a custom-written daemon).

...

What can be done to reduce its footprint?

Cut down on the databases. Are you using any 3rd party databases (sanesecurity for instance) ?

I'm using defaults, I think. Yes, the config file is dated dec 27 2016, so it is the installation file. I do not find any "sane" string in the config.

...

Instead of the daemon, I could call it from amavis, one process per mail. But this impacts scan speed. Could the daemon be started "on demand", and be killed a minute after the last email goes through?

Perhaps, but you're looking to save 500Mb on a machine with 8Gb?

I notice those 500, believe me. The machine responds a bit better just after I killed clamd.

Looking at your 'top' display, where does the rest of those 7Gb go?

Many things, it is just that clamav is the biggest chunk except for firefox and thunderbird. KiB Mem: 8174408 total, 4992444 used, 3181964 free, 103452 buffers KiB Swap: 43744244 total, 3190824 used, 40553420 free. 1909944 cached Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 20581 cer 20 0 4573608 1,064g 96912 416792 S 2,970 13,65 164:33.60 firefox 9866 cer 20 0 3045056 876360 73076 61684 S 0,000 10,72 25:26.60 thunderbird-bin 19773 cer 20 0 6670680 437464 41680 166220 S 0,330 5,352 10:14.16 soffice.bin 3874 root 20 0 1219596 161992 94228 234648 S 1,650 1,982 198:42.50 X 13111 cer 20 0 5270092 52544 9988 56176 S 0,330 0,643 0:22.19 java 4176 cer 20 0 754416 51876 18528 25848 S 0,000 0,635 3:11.76 xfdesktop 31344 cer 20 0 206044 33584 4936 53340 S 0,000 0,411 2:26.57 alpine 4210 cer 20 0 607528 29500 9720 39220 S 1,320 0,361 48:43.08 xfce4-terminal 4345 cer 20 0 509716 28680 16732 17996 S 0,000 0,351 0:04.38 Thunar 32676 vscan 20 0 166836 27992 5540 33220 S 0,000 0,342 0:00.30 /usr/sbin/amavi 17649 cer 20 0 424908 27712 6244 17432 S 0,000 0,339 0:28.00 kwalletd 4149 cer 20 0 510288 23400 9204 34344 S 0,000 0,286 9:15.91 xfce4-panel 3370 named 20 0 436824 19368 252 43828 S 0,000 0,237 3:41.93 named 4352 cer 39 19 1080232 18360 3108 13396 S 0,000 0,225 3:38.41 tracker-miner-f After that, the "RES" column has less than 20000 (clamd is manually stopped). Clamd has a bigger mem print than soffice.bin Notice that I have 3 G in swap. I would also benefit by using a very fast swap disk. That's another thing: I noticed that Leap reads swap very very slow. It can take two or four minutes for big apps (firefox, Libreoffice) to start responding after hibernation. 13.1 was much faster in this, almost instant response. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

On 2017-08-07 12:56, Wolfgang Rosenauer wrote:

Hi,

Am 07.08.2017 um 12:42 schrieb Carlos E. R.:

...

Many things, it is just that clamav is the biggest chunk except for firefox and thunderbird.

just to throw that in. Antivirus/Antispam is a quite expensive (as in memory, cpu and maintenance) function. As a mailserver admin this is something which is somehow expected.

Yes, that I understand.

If you run clamav on an extremely low traffic mailserver there still should be the option to run it on demand e.g. from amavisd-new. Have you looked into that option?

clamd is installed in the same clamav rpm, so it can't be removed. If clamd is not running, amavis tries insistently to reach it a few times before giving up and calling clamav or clamscan directly as a separate process. This is repeated for each mail, I think. And of course, starting clamav once per mail is heavy on a fetchmail run, which can pick a thousand emails in few minutes. I do not see how to configure amavis for not using clamd. If I start amavis, it starts clamd as a dependency. The best idea seems to be to configure systemd to start clamd by socket and stop it by timer. I need a howto for dummies on this. :-) -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

On 2017-08-07 13:24, Wolfgang Rosenauer wrote:

Am 07.08.2017 um 13:05 schrieb Carlos E. R.:

...

...

If you run clamav on an extremely low traffic mailserver there still should be the option to run it on demand e.g. from amavisd-new. Have you looked into that option?

clamd is installed in the same clamav rpm, so it can't be removed. If clamd is not running, amavis tries insistently to reach it a few times before giving up and calling clamav or clamscan directly as a separate process. This is repeated for each mail, I think. And of course, starting clamav once per mail is heavy on a fetchmail run, which can pick a thousand emails in few minutes.

I do not see how to configure amavis for not using clamd. If I start amavis, it starts clamd as a dependency.

Really? It used to be the way that you were able to comment the ClamAV-clamd entry in /etc/amavisd.conf and instead activate the clamscan in the same file.

I tried... either I did it wrong, or did not work. # ### http://www.clamav.net/ #['ClamAV-clamd', # \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd-socket"], # qr/\bOK$/m, qr/\bFOUND$/m, # qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ], I commented out that section.

On the other hand I see a Wants=named.service clamd.service which does not allow that setup :-(

Yes, that's what I supposed.

I think this hard requirement of Wants=clamd.service is actually too much in any case.

I tend to remove clamd.service from Wants unless someone has a smarter idea. Carlos, you can try locally as well if you like.

It is a possibility, yes.

...

The best idea seems to be to configure systemd to start clamd by socket and stop it by timer. I need a howto for dummies on this. :-)

I'm not a systemd expert on it. clamav is not my package but amavisd-new is somehow. An optional socket activation actually sounds useful in any case. Please ask the maintainer or via bugzilla explicitely what he thinks about it.

My idea was to find out how to do it, here, experiment, then suggest the optional improvement to the maintainer. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

On 2017-08-07 14:44, Per Jessen wrote:

Carlos E. R. wrote:

...

...

I think this hard requirement of Wants=clamd.service is actually too much in any case.

I tend to remove clamd.service from Wants unless someone has a smarter idea. Carlos, you can try locally as well if you like.

It is a possibility, yes.

Seems to be the easiest way out - provided you can get amavis to auto-start clamd:

in /etc/systemd/system/clamd.service.d/carlos.conf:

[Service] Wants=

...

My idea was to find out how to do it, here, experiment, then suggest the optional improvement to the maintainer.

A pretty long and comprehensive page: https://www.freedesktop.org/software/systemd/man/systemd.socket.html

Thanks, I'll read that.

Maybe there already examples of socket activated apps out there?

I guess so... -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

Carlos E. R. wrote:

On 2017-08-07 14:52, Carlos E. R. wrote:

...

...

...

My idea was to find out how to do it, here, experiment, then suggest the optional improvement to the maintainer.

A pretty long and comprehensive page: https://www.freedesktop.org/software/systemd/man/systemd.socket.html

Thanks, I'll read that.

Oh. It is a man page :-( List of options, no examples.

Yeah. The reference. On Leap422, I looked for examples - maybe look at avahi-daemon, that looks good. Not sure about the automatic shutdown though. -- Per Jessen, Zürich (23.3°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

On 2017-08-07 08:33, Per Jessen wrote:

...

Cut down on the databases. Are you using any 3rd party databases (sanesecurity for instance) ?

I'm using defaults, I think. Yes, the config file is dated dec 27 2016, so it is the installation file. I do not find any "sane" string in the config.

It wouldn't be in the config, clamd will load everything in the database dir. If you haven't added sanesecurity yourself, they're not there. You would also need a regular refresh etc. So it's probably just the default install - I don't have any historical data, but I have had clamd running on physical machines with 768Mb for years, now recently virtualised with about 1Gb each.

...

...

Instead of the daemon, I could call it from amavis, one process per mail. But this impacts scan speed. Could the daemon be started "on demand", and be killed a minute after the last email goes through?

Perhaps, but you're looking to save 500Mb on a machine with 8Gb?

I notice those 500, believe me. The machine responds a bit better just after I killed clamd.

Perhaps, but there must other processes gobbling up the rest, that's what I meant. The 500Mb for clamd is normal.

...

Looking at your 'top' display, where does the rest of those 7Gb go?

Many things, it is just that clamav is the biggest chunk except for firefox and thunderbird.

KiB Mem: 8174408 total, 4992444 used, 3181964 free, 103452 buffers KiB Swap: 43744244 total, 3190824 used, 40553420 free. 1909944 cached Mem

8Gb physical, 5Gb used, 3Gb swap and 2Gb file system buffers. Firefox and Thunderbird are also the biggest consumers on my main desktop, following by Xorg, konqueror and soffice. The rest less than 50Mb resident per process. Total: Mem: 4054784k total, 2976472k used, 1078312k free, 16k buffers Swap: 3911736k total, 778436k used, 3133300k free, 1113848k cached

Notice that I have 3 G in swap. I would also benefit by using a very fast swap disk.

Just add more memory, but you said you can't. Unless we're talking video editing and gaming (or something else that gobbles of memory), 4Gb should be more than plenty. Well, it is on my main desktop - not running Leap though. -- Per Jessen, Zürich (20.4°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

On 2017-08-07 13:20, Per Jessen wrote:

...

Carlos E. R. wrote:

...

On 2017-08-07 08:33, Per Jessen wrote:

...

Cut down on the databases. Are you using any 3rd party databases (sanesecurity for instance) ?

I'm using defaults, I think. Yes, the config file is dated dec 27 2016, so it is the installation file. I do not find any "sane" string in the config.

It wouldn't be in the config, clamd will load everything in the database dir. If you haven't added sanesecurity yourself, they're not there.

No such package here.

it's not packaged, sanesecurity maintains 3rd party signatures for clamav: http://sanesecurity.com/ I expect there are other databases out there too.

...

Perhaps, but there must other processes gobbling up the rest, that's what I meant. The 500Mb for clamd is normal.

I know that it is normal, for clamd. Still, it is a lot, it is the third biggest process in my machine. Doing nothing.

Does it do any regular scanning of your files? Just thinking out loud.

...

...

...

Looking at your 'top' display, where does the rest of those 7Gb go?

Many things, it is just that clamav is the biggest chunk except for firefox and thunderbird.

KiB Mem: 8174408 total, 4992444 used, 3181964 free, 103452 buffers KiB Swap: 43744244 total, 3190824 used, 40553420 free. 1909944 cached Mem

8Gb physical, 5Gb used, 3Gb swap and 2Gb file system buffers.

And 3 free.

Right, 8 - 5 = 3. Under those conditions, it's difficult to imagine 500Mb more or less free memory should make any noticable difference. You can try it yourself - int main( int argc, char **argv ) { int sz, p; char *f; sz=512*1024*1024L; f=malloc(sz); while( 1 ) { for( p=0; p

...

Just add more memory, but you said you can't. Unless we're talking video editing and gaming (or something else that gobbles of memory), 4Gb should be more than plenty. Well, it is on my main desktop - not running Leap though.

Leap seems to be very slow when swapping. The same machine with 13.1 was noticeably faster.

You said that before, but in relation to resuming from hibernation. That's not quite the same. Besides, your machine is not swapping?

I like to have all the applications I use open in a separate workspace, with some unfinished task, so I go back to them later.

Me too.

Thus, firefox, thunderbid and LO cn be opened at the same time. There is a java process that controls my TV.

Same here, except for the java. -- Per Jessen, Zürich (21.2°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote: [snip]

KiB Mem: 8174408 total, 5608116 used, 2566292 free, 207024 buffers KiB Swap: 43744244 total, 3097036 used, 40647208 free. 2419792 cached Mem

was:

KiB Mem: 8174408 total, 4992444 used, 3181964 free, 103452 buffers KiB Swap: 43744244 total, 3190824 used, 40553420 free. 1909944 cached Mem

...

...

...

Just add more memory, but you said you can't. Unless we're talking video editing and gaming (or something else that gobbles of memory), 4Gb should be more than plenty. Well, it is on my main desktop - not running Leap though.

Leap seems to be very slow when swapping. The same machine with 13.1 was noticeably faster.

You said that before, but in relation to resuming from hibernation. That's not quite the same. Besides, your machine is not swapping?

Yes, 3 GB in swap. See it has gone up during the morning.

"swapping" is more the situation where there is no or very little free memory. What you're seeing is more "has swapped in the past". If it were my system, I would look at that first. An 8Gb system with some 3Gb swap in use. Unless you know that you're running it to the max. None of our desktops have more than 4Gb memory, but they're not running Leap either. I've just had to re-allocate my main test system, I'll have to get another started. -- Per Jessen, Zürich (23.2°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

Remember this, when we wrote about the large memory footprint of clamd?

Well, it no longer happens, clamd is swapped out soon after a mail fetch run:

top - 19:33:22 up 11 days, 7:47, 1 user, load average: 0,44, 0,58, 0,93 Tasks: 520 total, 1 running, 517 sleeping, 0 stopped, 2 zombie %Cpu(s): 1,6 us, 0,6 sy, 0,0 ni, 96,6 id, 1,2 wa, 0,0 hi, 0,1 si, 0,0 st KiB Mem: 8174404 total, 2576192 used, 5598212 free, 45080 buffers KiB Swap: 25165820 total, 3958492 used, 21207328 free. 1152372 cached Mem

PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 3884 vscan 20 0 1054360 15528 0 519464 S 0,000 0,190 6:17.80 clamd 7269 cer 20 0 4083836 490408 53260 334432 S 0,331 5,999 250:57.99 thunderbird-bin

(sorted by swap usage)

I did nothing in the end. Perhaps some update altered the way clamd behaves regarding memory :-?

Unlikely I would say. Besides, it isn't really the job for an application to determine when to be swapped and when to be out :-) Your situation does look weird - about 5Gb of free memory, so although clam isn't in use at the moment, there's no reason for it to be swapped out. -- Per Jessen, Zürich (1.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sunday, 2017-12-17 at 12:15 +0100, Carlos E. R. wrote:

On Sunday, 2017-12-17 at 10:02 +0100, Per Jessen wrote:

...

I'll try to verify now, fresh tea in hand.

The file to look at would be /etc/amavisd.conf. It is dated Aug 11; this thread was in August, so it matches. And yes, the section seems commented out, so I'll undo.

No, it is not that. I just compared with file from rpm and the section was untouched. Still, clamd seems to be not working, not scanning mail (judging from 'top' cpu view). - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlo2V6kACgkQtTMYHG2NR9VxHQCeM/+r2lCffZDI8vf/RfzQ/yqr ovwAniU0SsM2DgeWyErzIw4CEbCyTVAL =PfoU -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org