[opensuse] Clamav process is now even more memory hungry
Photo using top, sort by memory, this instant:
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21395 cer 20 0 3402620 782716 85432 166280 S 69,35 9,591 61:58.83 Web Content 1096 cer 20 0 3688736 767644 104104 0 S 2,976 9,406 19:05.04 thunderbird-bin 3938 vscan 20 0 999648 725460 2628 17856 S 0,000 8,889 8:09.22 clamd .................................*******
0.7 GB of RAM! I will have to uninstall it. Las time I looked, it was half a gigabyte. And it is set with swapiness of 100... -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. wrote:
Photo using top, sort by memory, this instant:
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21395 cer 20 0 3402620 782716 85432 166280 S 69,35 9,591 61:58.83 Web Content 1096 cer 20 0 3688736 767644 104104 0 S 2,976 9,406 19:05.04 thunderbird-bin 3938 vscan 20 0 999648 725460 2628 17856 S 0,000 8,889 8:09.22 clamd .................................*******
0.7 GB of RAM! I will have to uninstall it. Las time I looked, it was half a gigabyte.
And it is set with swapiness of 100...
A couple of comments - increasing swappiness will only create more work for your system and increase latency in processing. - amount of memory used depends on the signature files you are using and somewhat on whether you are on 32bit or 64bit. On 32bit, I see 700Mb used, on 64bit I see 1Gb. - the amount memory used by clamd will continue to grow, albeit slowly. Options - - as you propose, don't use clamd, just live with the risk. - run clamscan on demand instead, if you don't care much about the latency and the extra work. - run clamd on a another machine - limit the signature files you are using or reduce their size. I don't know how feasible the latter is, but signatures in the database do have timestamps. -- Per Jessen, Zürich (19.8°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/09/2019 09.38, Per Jessen wrote:
Carlos E. R. wrote:
Photo using top, sort by memory, this instant:
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21395 cer 20 0 3402620 782716 85432 166280 S 69,35 9,591 61:58.83 Web Content 1096 cer 20 0 3688736 767644 104104 0 S 2,976 9,406 19:05.04 thunderbird-bin 3938 vscan 20 0 999648 725460 2628 17856 S 0,000 8,889 8:09.22 clamd .................................*******
0.7 GB of RAM! I will have to uninstall it. Las time I looked, it was half a gigabyte.
And it is set with swapiness of 100...
A couple of comments
- increasing swappiness will only create more work for your system and increase latency in processing.
Which is totally acceptable. It is already set to 100 for that process, and still it does not swap out on its own :-( Current status, after coming from hibernation this morning: PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 3938 vscan 20 0 999648 28740 2924 715764 S 0,000 0,352 8:09.72 clamd As you can see, now is almost totally swaped out. No issues. It just increased the used time some centiseconds. I will now send an email to myself on another computer, and check. <2.6> 2019-09-01 14:08:45 Telcontar postfix 4031 - - 482C33213B5: from=<cer@Telcontar.valinor>, size=892, nrcpt=1 (queue active) <2.6> 2019-09-01 14:08:45 Telcontar amavis 30621 - - (30621-03) ho0PUkymM_pr FWD from <cer@Telcontar.valinor> -> <cer@isengard.valinor>, BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 482C33213B5 <2.5> 2019-09-01 14:08:45 Telcontar amavis 30621 - - (30621-03) Passed CLEAN {RelayedInbound}, [127.0.0.1] <cer@telcontar.valinor> -> <cer@isengard.valinor>, Message-ID: <20190901120845.19CD43213B4@telcontar.valinor>, mail_id: ho0PUkymM_pr, Hits: -, size: 455, queued_as: 482C33213B5, 148 ms <2.6> 2019-09-01 14:08:45 Telcontar postfix 6145 - - 19CD43213B4: to=<cer@isengard.valinor>, orig_to=<cer@isengard>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.21, delays=0.02/0.04/0/0.15, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 482C33213B5) <2.6> 2019-09-01 14:08:45 Telcontar postfix 4031 - - 19CD43213B4: removed <2.6> 2019-09-01 14:08:45 Telcontar postfix 4031 - - 19CD43213B4: removed <2.6> 2019-09-01 14:08:45 Telcontar postfix 6150 - - 482C33213B5: to=<cer@isengard.valinor>, relay=isengard.valinor[192.168.1.16]:25, delay=0.21, delays=0/0.04/0.11/0.05, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 796FBA2131) As you can see, no impact at all. PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 3938 vscan 20 0 999648 28892 2924 715596 S 0,000 0,354 8:09.73 clamd Same thing. One centisecond more, almost same swap amount. Maybe clamd is only used on mail receive, and that is a manual operation when I call fetchmail. Maybe the clamd daemon is awakened periodically when the database is freshened. I do not understand why with a swapiness of 100 for that process, it doesn't swap out when it is not being used for hours. :-(
- amount of memory used depends on the signature files you are using and somewhat on whether you are on 32bit or 64bit. On 32bit, I see 700Mb used, on 64bit I see 1Gb.
- the amount memory used by clamd will continue to grow, albeit slowly.
:-(
Options -
- as you propose, don't use clamd, just live with the risk.
Which is probably nil, being on Linux. I can manually scan attachments.
- run clamscan on demand instead, if you don't care much about the latency and the extra work.
I failed at configuring this.
- run clamd on a another machine
I failed at doing this. No idea how to do it, unless I move the entire amavis. The other machine has free memory but the CPU is way less powerful.
- limit the signature files you are using or reduce their size. I don't know how feasible the latter is, but signatures in the database do have timestamps.
The only thing that worries me would be scanning PDF and other document types. Win executables I don't expect any and amavis can handle them on its own. If it were possible to unload all exe signatures... Or perhaps there is another antivirus we can use :-? -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. wrote:
On 01/09/2019 09.38, Per Jessen wrote:
Carlos E. R. wrote:
Photo using top, sort by memory, this instant:
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21395 cer 20 0 3402620 782716 85432 166280 S 69,35 9,591 61:58.83 Web Content 1096 cer 20 0 3688736 767644 104104 0 S 2,976 9,406 19:05.04 thunderbird-bin 3938 vscan 20 0 999648 725460 2628 17856 S 0,000 8,889 8:09.22 clamd .................................*******
0.7 GB of RAM! I will have to uninstall it. Las time I looked, it was half a gigabyte.
And it is set with swapiness of 100...
A couple of comments
- increasing swappiness will only create more work for your system and increase latency in processing.
Which is totally acceptable. It is already set to 100 for that process, and still it does not swap out on its own :-(
In principle, the memory could be pinned, but I see no mlock* calls in libclamav. I don't know if there are other ways though. maybe clamd does a regular traverse across the signature database, dunno.
Current status, after coming from hibernation this morning:
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND
3938 vscan 20 0 999648 28740 2924 715764 S 0,000 0,352 8:09.72 clamd
Does hibernation somehow force processes to swap out?
As you can see, now is almost totally swaped out. No issues. It just increased the used time some centiseconds. I will now send an email to myself on another computer, and check. [snip]>
As you can see, no impact at all.
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND
3938 vscan 20 0 999648 28892 2924 715596 S 0,000 0,354 8:09.73 clamd
Same thing. One centisecond more, almost same swap amount.
Your email was not scanned ?
Maybe clamd is only used on mail receive, and that is a manual operation when I call fetchmail. Maybe the clamd daemon is awakened periodically when the database is freshened.
Definitely the latter - the database is often updated a few times a day. I guess you are running freshclam?
I do not understand why with a swapiness of 100 for that process, it doesn't swap out when it is not being used for hours. :-(
If nothing needs to use that memory?
- run clamd on a another machine
I failed at doing this. No idea how to do it, unless I move the entire amavis. The other machine has free memory but the CPU is way less powerful.
My old test system cluster ran on Pentium II 450MHz, it did just fine. clamd can be configured to listen for external connections, now you just need to make amavis talk to an external clamd. -- Per Jessen, Zürich (21.8°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/09/2019 14.51, Per Jessen wrote:
Carlos E. R. wrote:
On 01/09/2019 09.38, Per Jessen wrote:
Carlos E. R. wrote:
Photo using top, sort by memory, this instant:
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21395 cer 20 0 3402620 782716 85432 166280 S 69,35 9,591 61:58.83 Web Content 1096 cer 20 0 3688736 767644 104104 0 S 2,976 9,406 19:05.04 thunderbird-bin 3938 vscan 20 0 999648 725460 2628 17856 S 0,000 8,889 8:09.22 clamd .................................*******
0.7 GB of RAM! I will have to uninstall it. Las time I looked, it was half a gigabyte.
And it is set with swapiness of 100...
A couple of comments
- increasing swappiness will only create more work for your system and increase latency in processing.
Which is totally acceptable. It is already set to 100 for that process, and still it does not swap out on its own :-(
In principle, the memory could be pinned, but I see no mlock* calls in libclamav. I don't know if there are other ways though. maybe clamd does a regular traverse across the signature database, dunno.
No, it is still swapped out, an hour later. It is the data segment, not the code.
Current status, after coming from hibernation this morning:
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND
3938 vscan 20 0 999648 28740 2924 715764 S 0,000 0,352 8:09.72 clamd
Does hibernation somehow force processes to swap out?
Yes, fully. Apparently they are all frozen, no cpu time; then discardable parts are just released, the rest is swapped out to normal swap, so to say. It is not a linear dump of all memory, but app by app - or at least, first app by app, then a dump of what remains. Educated guess.
As you can see, now is almost totally swaped out. No issues. It just increased the used time some centiseconds. I will now send an email to myself on another computer, and check. [snip]>
As you can see, no impact at all.
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND
3938 vscan 20 0 999648 28892 2924 715596 S 0,000 0,354 8:09.73 clamd
Same thing. One centisecond more, almost same swap amount.
Your email was not scanned ?
It was, see the amavis entry in the log. And headers on the received mail: Received: from localhost (localhost [127.0.0.1]) by Telcontar.valinor (Postfix) with ESMTP id 482C33213B5 for <cer@isengard.valinor>; Sun, 1 Sep 2019 14:08:45 +0200 (CEST) X-Virus-Scanned: amavisd-new at valinor Received: from Telcontar.valinor ([127.0.0.1]) by localhost (telcontar.valinor [127.0.0.1]) (amavisd-new, port 10024) with LMTP id ho0PUkymM_pr for <cer@isengard.valinor>; Sun, 1 Sep 2019 14:08:45 +0200 (CEST) I can try the reverse instead, sending to this machine. [...] Done. It was scanned: X-Virus-Scanned: amavisd-new at valinor Let's see the log: <2.6> 2019-09-01 15:30:23 Telcontar postfix 4031 - - 83A653213B5: from=<cer@Isengard.valinor>, size=1459, nrcpt=1 (queue active) <2.6> 2019-09-01 15:30:23 Telcontar amavis 30621 - - (30621-08) HuP8u8ciVXg9 FWD from <cer@Isengard.valinor> -> <cer@Telcontar.valinor>, BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 83A653213B5 <2.6> 2019-09-01 15:30:23 Telcontar postfix 10745 - - disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 <2.5> 2019-09-01 15:30:23 Telcontar amavis 30621 - - (30621-08) Passed CLEAN {RelayedInternal}, MYNETS LOCAL [192.168.1.16]:45176 <cer@isengard.valinor> -> <cer@telcontar.valinor>, Queue-ID: 5901D3213AD, Message-ID: <alpine.LSU.2.21.1909011530130.4300@isengard.valinor>, mail_id: HuP8u8ciVXg9, Hits: -, size: 1020, queued_as: 83A653213B5, 138 ms <2.6> 2019-09-01 15:30:23 Telcontar postfix 10742 - - 5901D3213AD: to=<cer@Telcontar.valinor>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.19, delays=0.01/0.04/0/0.14, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 83A653213B5) All within a second. It is amavis who calls clamd. Perhaps it doesn't if there are no attachments. PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 3938 vscan 20 0 999648 34304 2924 710216 S 0,000 0,420 8:10.06 clamd It used less than a second CPU time in the last hour. -rw-r--r-- 1 root root 981 May 13 2018 /usr/share/cups/data/confidential.pdf I'll mail that to me. <2.6> 2019-09-01 15:36:39 Telcontar postfix 4031 - - 3B0AA3213B6: from=<cer@Isengard.valinor>, size=3430, nrcpt=1 (queue active) <2.6> 2019-09-01 15:36:39 Telcontar amavis 29786 - - (29786-09) ABdPf5OUOzE0 FWD from <cer@Isengard.valinor> -> <cer@Telcontar.valinor>, BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 3B0AA3213B6 <2.5> 2019-09-01 15:36:39 Telcontar amavis 29786 - - (29786-09) Passed CLEAN {RelayedInternal}, MYNETS LOCAL [192.168.1.16]:45278 <cer@isengard.valinor> -> <cer@telcontar.valinor>, Queue-ID: F12803213AD, Message-ID: <alpine.LSU.2.21.1909011535510.4807@isengard.valinor>, mail_id: ABdPf5OUOzE0, Hits: -, size: 2991, queued_as: 3B0AA3213B6, 216 ms <2.6> 2019-09-01 15:36:39 Telcontar postfix 11049 - - F12803213AD: to=<cer@Telcontar.valinor>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.27, delays=0.01/0.04/0/0.22, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 3B0AA3213B6) PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 3938 vscan 20 0 999648 37720 3832 708536 S 0,000 0,462 8:10.10 clamd I'll have to mail myself a sample virus. [...] Detected: 2.6> 2019-09-01 15:46:46 Telcontar postfix 11639 - - disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 <2.6> 2019-09-01 15:46:46 Telcontar amavis 30621 - - (30621-09) bn83MTXvr3CP FWD from <cer@Isengard.valinor> -> <cer+virus@Telcontar.valinor>, BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 530BE3213B7 <2.5> 2019-09-01 15:46:46 Telcontar amavis 30621 - - (30621-09) Passed INFECTED (Win.Worm.N-74) {RelayedTaggedInternal,Quarantined}, MYNETS LOCAL [192.168.1.16]:46054 <cer@isengard.valinor> -> <cer@telcontar.valinor>, quarantine: virus-bn83MTXvr3CP, Queue-ID: 67BD73213AD, Message-ID: <alpine.LSU.2.21.1909011546180.4807@isengard.valinor>, mail_id: bn83MTXvr3CP, Hits: -, size: 407964, queued_as: 530BE3213B7, 3919 ms <2.6> 2019-09-01 15:46:46 Telcontar postfix 11632 - - 67BD73213AD: to=<cer@Telcontar.valinor>, relay=127.0.0.1[127.0.0.1]:10024, delay=4, delays=0.02/0.04/0/3.9, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 530BE3213B7) <2.6> 2019-09-01 15:46:46 Telcontar postfix 4031 - - 67BD73213AD: removed <2.6> 2019-09-01 15:46:47 Telcontar postfix 11649 - - 530BE3213B7: to=<cer+virus@Telcontar.valinor>, relay=local, delay=0.72, delays=0.06/0.01/0/0.65, dsn=2.0.0, status=sent (delivered to command: /usr/bin/procmail) <2.6> 2019-09-01 15:46:47 Telcontar postfix 4031 - - 530BE3213B7: removed <2.6> 2019-09-01 15:46:47 Telcontar dovecot - - - lda(cer)<11657><I/K7BMfLa12JLQAAoyW3yA>: msgid=<VAbn83MTXvr3CP@telcontar.valinor>: saved mail to in_root <2.6> 2019-09-01 15:46:47 Telcontar postfix 11640 - - 4D3713213B6: to=<cer@Telcontar.valinor>, orig_to=<virusalert@telcontar.valinor>, relay=local, delay=0.94, delays=0.01/0.01/0/0.92, dsn=2.0.0, status=sent (delivered to command: /usr/bin/procmail) <2.6> 2019-09-01 15:46:47 Telcontar postfix 4031 - - 4D3713213B6: removed One second to process. And, surprise! See memory use: PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 3938 vscan 20 0 999648 121692 3832 624804 S 0,000 1,491 8:11.65 clamd One second and a half CPU time used, less than a 100MB RES increase. So you see, clamd can remain swapped out most of the time in my environment with no impairment.
Maybe clamd is only used on mail receive, and that is a manual operation when I call fetchmail. Maybe the clamd daemon is awakened periodically when the database is freshened.
Definitely the latter - the database is often updated a few times a day. I guess you are running freshclam?
Yes. That would certainly cause the process to be in RAM, but after a while it should be swapped out again.
I do not understand why with a swapiness of 100 for that process, it doesn't swap out when it is not being used for hours. :-(
If nothing needs to use that memory?
But it does, swap is in use all the time, the system I noticed a bit slow yesterday. Swaping out clamd would release some RAM, better used by other processes. And I'm god, I ordered swapiness 100 for that process. I decide. Sigh... it does not obey. Why? cer@Telcontar:/sys/fs/cgroup/memory/clamd> l total 0 drwxr-xr-x 2 root root 0 Sep 1 15:57 ./ dr-xr-xr-x 3 root root 0 Sep 1 15:55 ../ -rw-r--r-- 1 root root 0 Sep 1 15:57 cgroup.clone_children --w--w--w- 1 root root 0 Sep 1 15:57 cgroup.event_control -rw-r--r-- 1 root root 0 Aug 26 11:50 cgroup.procs -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.failcnt --w------- 1 root root 0 Sep 1 15:57 memory.force_empty -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.kmem.failcnt -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.kmem.limit_in_bytes -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.kmem.max_usage_in_bytes -r--r--r-- 1 root root 0 Sep 1 15:57 memory.kmem.slabinfo -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.kmem.tcp.failcnt -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.kmem.tcp.limit_in_bytes -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.kmem.tcp.max_usage_in_bytes -r--r--r-- 1 root root 0 Sep 1 15:57 memory.kmem.tcp.usage_in_bytes -r--r--r-- 1 root root 0 Sep 1 15:57 memory.kmem.usage_in_bytes -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.limit_in_bytes -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.max_usage_in_bytes -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.move_charge_at_immigrate -r--r--r-- 1 root root 0 Sep 1 15:57 memory.numa_stat -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.oom_control ---------- 1 root root 0 Sep 1 15:57 memory.pressure_level -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.soft_limit_in_bytes -r--r--r-- 1 root root 0 Sep 1 15:57 memory.stat -rw-r--r-- 1 root root 0 Aug 26 11:49 memory.swappiness -r--r--r-- 1 root root 0 Sep 1 15:57 memory.usage_in_bytes -rw-r--r-- 1 root root 0 Sep 1 15:57 memory.use_hierarchy -rw-r--r-- 1 root root 0 Sep 1 15:57 notify_on_release -rw-r--r-- 1 root root 0 Sep 1 15:57 tasks cer@Telcontar:/sys/fs/cgroup/memory/clamd> cer@Telcontar:/sys/fs/cgroup/memory/clamd> cat memory.swappiness 100 cer@Telcontar:/sys/fs/cgroup/memory/clamd>
- run clamd on a another machine
I failed at doing this. No idea how to do it, unless I move the entire amavis. The other machine has free memory but the CPU is way less powerful.
My old test system cluster ran on Pentium II 450MHz, it did just fine. clamd can be configured to listen for external connections, now you just need to make amavis talk to an external clamd.
I don't know how to do that. Do you have a link? -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. wrote:
On 01/09/2019 14.51, Per Jessen wrote:
Carlos E. R. wrote:
On 01/09/2019 09.38, Per Jessen wrote:
Carlos E. R. wrote:
Photo using top, sort by memory, this instant:
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21395 cer 20 0 3402620 782716 85432 166280 S 69,35 9,591 61:58.83 Web Content 1096 cer 20 0 3688736 767644 104104 0 S 2,976 9,406 19:05.04 thunderbird-bin 3938 vscan 20 0 999648 725460 2628 17856 S 0,000 8,889 8:09.22 clamd .................................*******
0.7 GB of RAM! I will have to uninstall it. Las time I looked, it was half a gigabyte.
And it is set with swapiness of 100...
A couple of comments
- increasing swappiness will only create more work for your system and increase latency in processing.
Which is totally acceptable. It is already set to 100 for that process, and still it does not swap out on its own :-(
In principle, the memory could be pinned, but I see no mlock* calls in libclamav. I don't know if there are other ways though. maybe clamd does a regular traverse across the signature database, dunno.
No, it is still swapped out, an hour later. It is the data segment, not the code.
Yes, it's the signature databases.
As you can see, now is almost totally swaped out. No issues. It just increased the used time some centiseconds. I will now send an email to myself on another computer, and check. [snip]>
As you can see, no impact at all.
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND
3938 vscan 20 0 999648 28892 2924 715596 S 0,000 0,354 8:09.73 clamd
Same thing. One centisecond more, almost same swap amount.
Your email was not scanned ?
It was, see the amavis entry in the log. And headers on the received mail:
Received: from localhost (localhost [127.0.0.1]) by Telcontar.valinor (Postfix) with ESMTP id 482C33213B5 for <cer@isengard.valinor>; Sun, 1 Sep 2019 14:08:45 +0200 (CEST) X-Virus-Scanned: amavisd-new at valinor
I don't believe that for one second. clamd works by loading all signatures into memory and using that when scanning. Maybe you could add "LogTime true" to clamd.conf ?
Let's see the log:
[snip]
All within a second. It is amavis who calls clamd. Perhaps it doesn't if there are no attachments.
That would make some sense, I think.
So you see, clamd can remain swapped out most of the time in my environment with no impairment.
I think there is something wrong - something is schewing the operation.
I do not understand why with a swapiness of 100 for that process, it doesn't swap out when it is not being used for hours. :-(
If nothing needs to use that memory?
But it does, swap is in use all the time, the system I noticed a bit slow yesterday. Swaping out clamd would release some RAM, better used by other processes. And I'm god, I ordered swapiness 100 for that process. I decide. Sigh... it does not obey. Why?
You have only decided that the clamd memory _can_ be swapped out, not that it _will_ be swapped out. There is no need to swap out if nothing else needs the memory. Or somehow your swappiness setting doesn't take effect?
- run clamd on a another machine
I failed at doing this. No idea how to do it, unless I move the entire amavis. The other machine has free memory but the CPU is way less powerful.
My old test system cluster ran on Pentium II 450MHz, it did just fine. clamd can be configured to listen for external connections, now you just need to make amavis talk to an external clamd.
I don't know how to do that. Do you have a link?
No, I don't do it. man clamd.conf and man amavisd.conf ? -- Per Jessen, Zürich (22.5°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/09/2019 16.52, Per Jessen wrote:
Carlos E. R. wrote:
On 01/09/2019 14.51, Per Jessen wrote:
Carlos E. R. wrote:
No, it is still swapped out, an hour later. It is the data segment, not the code.
Yes, it's the signature databases.
As you can see, now is almost totally swaped out. No issues. It just increased the used time some centiseconds. I will now send an email to myself on another computer, and check. [snip]>
As you can see, no impact at all.
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND
3938 vscan 20 0 999648 28892 2924 715596 S 0,000 0,354 8:09.73 clamd
Same thing. One centisecond more, almost same swap amount.
Your email was not scanned ?
It was, see the amavis entry in the log. And headers on the received mail:
Received: from localhost (localhost [127.0.0.1]) by Telcontar.valinor (Postfix) with ESMTP id 482C33213B5 for <cer@isengard.valinor>; Sun, 1 Sep 2019 14:08:45 +0200 (CEST) X-Virus-Scanned: amavisd-new at valinor
I don't believe that for one second. clamd works by loading all signatures into memory and using that when scanning. Maybe you could add "LogTime true" to clamd.conf ?
Well, you can see how it actually detected a virus, and not the entire thing came back to ram. I guess memory segments were retrieved as needed from swap by the kernel, till clamd hit a match signature and it stopped. Less than 100MB were needed. The setting is active, anyway: # Log time with each message. # Default: no LogTime yes
Let's see the log:
[snip]
All within a second. It is amavis who calls clamd. Perhaps it doesn't if there are no attachments.
That would make some sense, I think.
So you see, clamd can remain swapped out most of the time in my environment with no impairment.
I think there is something wrong - something is schewing the operation.
I do not understand why with a swapiness of 100 for that process, it doesn't swap out when it is not being used for hours. :-(
If nothing needs to use that memory?
But it does, swap is in use all the time, the system I noticed a bit slow yesterday. Swaping out clamd would release some RAM, better used by other processes. And I'm god, I ordered swapiness 100 for that process. I decide. Sigh... it does not obey. Why?
You have only decided that the clamd memory _can_ be swapped out, not that it _will_ be swapped out. There is no need to swap out if nothing else needs the memory. Or somehow your swappiness setting doesn't take effect?
You see I posted the sys entries that show it is active. Id like to find something more aggressive.
- run clamd on a another machine
I failed at doing this. No idea how to do it, unless I move the entire amavis. The other machine has free memory but the CPU is way less powerful.
My old test system cluster ran on Pentium II 450MHz, it did just fine. clamd can be configured to listen for external connections, now you just need to make amavis talk to an external clamd.
I don't know how to do that. Do you have a link?
No, I don't do it.
man clamd.conf and man amavisd.conf ?
I tried... but they are just a list of options and their syntax. Not how to do something. Clamd listen on a port, amavis could be redirected to that port outside. But the problem is, amavis autodetects clamd and connects to it, so connecting to outside means changing code, IMHO. # ### http://www.clamav.net/ ['ClamAV-clamd', \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd-socket"], qr/\bOK$/m, qr/\bFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ], # # NOTE: run clamd under the same user as amavisd - or run it under its own # # uid such as clamav, add user clamav to the amavis group, and then add # # AllowSupplementaryGroups to clamd.conf; # # NOTE: match socket name (LocalSocket) in clamav.conf to the socket name in # # this entry; when running chrooted one may prefer a socket under $MYHOME. cer@Telcontar:~> l /var/run/clamav/clamd-socket srw-rw-rw- 1 vscan vscan 0 Aug 26 11:50 /var/run/clamav/clamd-socket= Huh, not a port, but a socket. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. wrote:
I don't believe that for one second. clamd works by loading all signatures into memory and using that when scanning. Maybe you could add "LogTime true" to clamd.conf ?
Well, you can see how it actually detected a virus, and not the entire thing came back to ram. I guess memory segments were retrieved as needed from swap by the kernel, till clamd hit a match signature and it stopped. Less than 100MB were needed.
And the test signature is probably one of the first in the database, yes. I don't think the PDF you sent was scanned though.
The setting is active, anyway:
# Log time with each message. # Default: no LogTime yes
So there must be a log somewhere? That would be conclusive evidence.
But it does, swap is in use all the time, the system I noticed a bit slow yesterday. Swaping out clamd would release some RAM, better used by other processes. And I'm god, I ordered swapiness 100 for that process. I decide. Sigh... it does not obey. Why?
You have only decided that the clamd memory _can_ be swapped out, not that it _will_ be swapped out. There is no need to swap out if nothing else needs the memory. Or somehow your swappiness setting doesn't take effect?
You see I posted the sys entries that show it is active.
Yeah I saw that. Yet your experience seems to contradict it.
Id like to find something more aggressive.
Maybe try "kill -TSTP <pid>" to suspend/pause clamd.
man clamd.conf and man amavisd.conf ?
I tried... but they are just a list of options and their syntax. Not how to do something.
Well, no. That is the job for the sysadmin :-) You need to enable clamd for listening on an external socket - that seems to be the TCPSocket and the TCPAddr options. I don't have amavis installed anywhere, but I am sure you can find parameters that directs which clamd to use.
Clamd listen on a port, amavis could be redirected to that port outside. But the problem is, amavis autodetects clamd and connects to it, so connecting to outside means changing code, IMHO.
There is nothing to configure which clamd to use?
# ### http://www.clamav.net/ ['ClamAV-clamd', \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd-socket"], qr/\bOK$/m, qr/\bFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
Wild guess: replace "/var/run/clamav/clamd-socket"] with "yourhost:yourport" -- Per Jessen, Zürich (22.2°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/09/2019 18.33, Per Jessen wrote:
Carlos E. R. wrote:
I don't believe that for one second. clamd works by loading all signatures into memory and using that when scanning. Maybe you could add "LogTime true" to clamd.conf ?
Well, you can see how it actually detected a virus, and not the entire thing came back to ram. I guess memory segments were retrieved as needed from swap by the kernel, till clamd hit a match signature and it stopped. Less than 100MB were needed.
And the test signature is probably one of the first in the database, yes. I don't think the PDF you sent was scanned though.
I think the same, and that worries me. Unless it knows directly it has no javascript.
The setting is active, anyway:
# Log time with each message. # Default: no LogTime yes
So there must be a log somewhere? That would be conclusive evidence.
The mail log, as far I know, is the mail log, and I posted the entire relevant section. There is no other log file modified today: ls -ltr /var/log/ ... -rw-r----- 1 root root 5996841 Aug 31 16:53 mail.err -rw-r----- 1 root root 591116 Sep 1 00:00 allmessages-20190901.xz drwxr-xr-x 31 root root 8192 Sep 1 00:00 ./ -rw-r--r-- 1 root root 498220 Sep 1 13:30 Sesiones.log -rw-r--r-- 1 root root 65312 Sep 1 13:42 Xorg.0.log -rw-r--r-- 1 root root 918991 Sep 1 14:06 ntp -rw-r----- 1 root root 3113036 Sep 1 19:04 warn -rw-r----- 1 root root 4504412 Sep 1 19:04 mail.warn -rw-r----- 1 root root 2672662 Sep 1 19:04 named -rw-r----- 1 root root 3956350 Sep 1 20:01 snapper.log -rw-r----- 1 root root 2002551 Sep 1 20:06 mail -rw-r----- 1 root root 2001393 Sep 1 20:06 mail.info -rw-r----- 1 root root 3086591 Sep 1 20:40 messages -rw-r----- 1 root root 6338661 Sep 1 20:44 pruned -rw-r----- 1 root root 2662844 Sep 1 20:44 allmessages Ah, no: # Uncomment this option to enable logging. # LogFile must be writable for the user running daemon. # A full path is required. # Default: disabled #LogFile /tmp/clamd.log
But it does, swap is in use all the time, the system I noticed a bit slow yesterday. Swaping out clamd would release some RAM, better used by other processes. And I'm god, I ordered swapiness 100 for that process. I decide. Sigh... it does not obey. Why?
You have only decided that the clamd memory _can_ be swapped out, not that it _will_ be swapped out. There is no need to swap out if nothing else needs the memory. Or somehow your swappiness setting doesn't take effect?
You see I posted the sys entries that show it is active.
Yeah I saw that. Yet your experience seems to contradict it.
It is swapped out faster than what it was before I did that configuration change, but not as aggressively as I thought.
Id like to find something more aggressive.
Maybe try "kill -TSTP <pid>" to suspend/pause clamd.
Mmmm... :-? The process is already waiting, not using cpu at all.
man clamd.conf and man amavisd.conf ?
I tried... but they are just a list of options and their syntax. Not how to do something.
Well, no. That is the job for the sysadmin :-)
Haha.
You need to enable clamd for listening on an external socket - that seems to be the TCPSocket and the TCPAddr options.
I don't have amavis installed anywhere, but I am sure you can find parameters that directs which clamd to use.
I posted the part of the configuration that mentions clamd. Wait... I found something. @av_scanners_backup = ( ### http://www.clamav.net/ - backs up clamd or Mail::ClamAV ['ClamAV-clamscan', 'clamscan', "--stdout --no-summary -r --tempdir=$TEMPBASE {}", [0], qr/:.*\sFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ], # ### http://www.clamav.net/ - using remote clamd scanner as a backup # ['ClamAV-clamdscan', 'clamdscan', # "--stdout --no-summary --config-file=/etc/clamd-client.conf {}", # [0], qr/:.*\sFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
Clamd listen on a port, amavis could be redirected to that port outside. But the problem is, amavis autodetects clamd and connects to it, so connecting to outside means changing code, IMHO.
There is nothing to configure which clamd to use?
Not that I know, no. The config file is code.
# ### http://www.clamav.net/ ['ClamAV-clamd', \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd-socket"], qr/\bOK$/m, qr/\bFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
Wild guess:
replace "/var/run/clamav/clamd-socket"] with
"yourhost:yourport"
:-? It is an area I know little about. There is some mention in the clamd config, though: # Clamd uses FTP-like protocol to receive data from remote clients. # If you are using clamav-milter to balance load between remote clamd daemons # on firewall servers you may need to tune the options below. And there is saint google: "using amavis with a remote clamd" First hit, 15 years ago, says it is not currently supported. Second one, does not use amavis. Here there is one hit (2007): https://sourceforge.net/p/amavis/mailman/message/17392709/ ah, but he says it is not working. The answer is it is not possible. Clamav supports it, amavis not. Crumbs :-( -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. wrote:
On 01/09/2019 18.33, Per Jessen wrote:
Carlos E. R. wrote:
I don't believe that for one second. clamd works by loading all signatures into memory and using that when scanning. Maybe you could add "LogTime true" to clamd.conf ?
Well, you can see how it actually detected a virus, and not the entire thing came back to ram. I guess memory segments were retrieved as needed from swap by the kernel, till clamd hit a match signature and it stopped. Less than 100MB were needed.
And the test signature is probably one of the first in the database, yes. I don't think the PDF you sent was scanned though.
I think the same, and that worries me. Unless it knows directly it has no javascript.
I think it is unlikely for amavis to dabble with that.
Id like to find something more aggressive.
Maybe try "kill -TSTP <pid>" to suspend/pause clamd.
Mmmm... :-? The process is already waiting, not using cpu at all.
It can still be scheduled, receive data etc. If you pause it, it can't.
You need to enable clamd for listening on an external socket - that seems to be the TCPSocket and the TCPAddr options.
I don't have amavis installed anywhere, but I am sure you can find parameters that directs which clamd to use.
I posted the part of the configuration that mentions clamd. Wait... I found something. @av_scanners_backup = (
### http://www.clamav.net/ - backs up clamd or Mail::ClamAV ['ClamAV-clamscan', 'clamscan', "--stdout --no-summary -r --tempdir=$TEMPBASE {}", [0], qr/:.*\sFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m [],
That sounds a lot like the on-demand scan you were looking for?
Clamd listen on a port, amavis could be redirected to that port outside. But the problem is, amavis autodetects clamd and connects to it, so connecting to outside means changing code, IMHO.
There is nothing to configure which clamd to use?
Not that I know, no. The config file is code.
Okay, so it's configured with perl.
# ### http://www.clamav.net/ ['ClamAV-clamd', \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd-socket"], qr/\bOK$/m, qr/\bFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
Wild guess:
replace "/var/run/clamav/clamd-socket"] with
"yourhost:yourport"
:-?
It is an area I know little about.
If I read it correctly, that definition is called "ClamAV-clamd", it uses a function called "ask_daemon" to talk to a daemon listening on "/var/run/clamav/clamd-socket". The rest are regexes for how to interpret the result. Instead of a UNIX socket path, it seems reasonable to use "yourhost:yourport" instead.
And there is saint google: "using amavis with a remote clamd"
First hit, 15 years ago, says it is not currently supported. Second one, does not use amavis.
Here there is one hit (2007):
https://sourceforge.net/p/amavis/mailman/message/17392709/
ah, but he says it is not working. The answer is it is not possible. Clamav supports it, amavis not.
Oh. But there is an example in the config you posted? # ### http://www.clamav.net/ - using remote clamd scanner as a backup # ['ClamAV-clamdscan', 'clamdscan', # "--stdout --no-summary --config-file=/etc/clamd-client.conf {}", # [0], qr/:.*\sFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ], I would try it, if this memory issue is so critical. -- Per Jessen, Zürich (16.2°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/09/2019 07.58, Per Jessen wrote:
Carlos E. R. wrote:
On 01/09/2019 18.33, Per Jessen wrote:
Carlos E. R. wrote:
I don't believe that for one second. clamd works by loading all signatures into memory and using that when scanning. Maybe you could add "LogTime true" to clamd.conf ?
Well, you can see how it actually detected a virus, and not the entire thing came back to ram. I guess memory segments were retrieved as needed from swap by the kernel, till clamd hit a match signature and it stopped. Less than 100MB were needed.
And the test signature is probably one of the first in the database, yes. I don't think the PDF you sent was scanned though.
I think the same, and that worries me. Unless it knows directly it has no javascript.
I think it is unlikely for amavis to dabble with that.
Id like to find something more aggressive.
Maybe try "kill -TSTP <pid>" to suspend/pause clamd.
Mmmm... :-? The process is already waiting, not using cpu at all.
It can still be scheduled, receive data etc. If you pause it, it can't.
Right, but then it will not respond when a mail has to be scanned.
You need to enable clamd for listening on an external socket - that seems to be the TCPSocket and the TCPAddr options.
I don't have amavis installed anywhere, but I am sure you can find parameters that directs which clamd to use.
I posted the part of the configuration that mentions clamd. Wait... I found something. @av_scanners_backup = (
### http://www.clamav.net/ - backs up clamd or Mail::ClamAV ['ClamAV-clamscan', 'clamscan', "--stdout --no-summary -r --tempdir=$TEMPBASE {}", [0], qr/:.*\sFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m [],
That sounds a lot like the on-demand scan you were looking for?
Huh? Ah. That would be not using clamd, but clamav. It takes ages to start and load. cer@Telcontar:~/viruses> time clamscan sample.exe-virus_W32-Nimd sample.exe-virus_W32-Nimd: Win.Worm.N-74 FOUND ----------- SCAN SUMMARY ----------- Known viruses: 6303089 Engine version: 0.101.3 Scanned directories: 0 Scanned files: 1 Infected files: 1 Data scanned: 0.12 MB Data read: 0.28 MB (ratio 0.44:1) Time: 57.688 sec (0 m 57 s) real 0m57,729s user 0m57,116s sys 0m0,608s cer@Telcontar:~/viruses> One minute. If one second later I ask again, it is another minute. What I want is to start the clamd daemon on socket; take a minute to start, but then the following jobs would be done fast. After a time (say, 2 minutes) of no activities, unload.
Clamd listen on a port, amavis could be redirected to that port outside. But the problem is, amavis autodetects clamd and connects to it, so connecting to outside means changing code, IMHO.
There is nothing to configure which clamd to use?
Not that I know, no. The config file is code.
Okay, so it's configured with perl.
# ### http://www.clamav.net/ ['ClamAV-clamd', \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd-socket"], qr/\bOK$/m, qr/\bFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
Wild guess:
replace "/var/run/clamav/clamd-socket"] with
"yourhost:yourport"
:-?
It is an area I know little about.
If I read it correctly, that definition is called "ClamAV-clamd", it uses a function called "ask_daemon" to talk to a daemon listening on "/var/run/clamav/clamd-socket". The rest are regexes for how to interpret the result. Instead of a UNIX socket path, it seems reasonable to use "yourhost:yourport" instead.
And there is saint google: "using amavis with a remote clamd"
First hit, 15 years ago, says it is not currently supported. Second one, does not use amavis.
Here there is one hit (2007):
https://sourceforge.net/p/amavis/mailman/message/17392709/
ah, but he says it is not working. The answer is it is not possible. Clamav supports it, amavis not.
Oh. But there is an example in the config you posted?
# ### http://www.clamav.net/ - using remote clamd scanner as a backup # ['ClamAV-clamdscan', 'clamdscan', # "--stdout --no-summary --config-file=/etc/clamd-client.conf {}", # [0], qr/:.*\sFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
I would try it, if this memory issue is so critical.
But there is no documentation of this :-( It needs a config file, /etc/clamd-client.conf There is no clamd-client in my system. Maybe it is for clamdscan, but man page does not mention anything related. Google hits for "clamd-client.conf" mentions amavis only. Found something. <https://sourceforge.net/p/amavis/mailman/message/23962953/> +++................ Given the overhead of having to stream the whole content to a remote scanner, the cost of spawning a clamdscan process is probably negligible. I don't think there is a need to duplicate in amavisd what clamdscan client does just fine.
if clamd supported both TCP and unix sockets on the same server, and you had two servers, amavisd could use unix sockets for primary, and then maybe tcp for backup scanner.
Just add an entry like: ### http://www.clamav.net/ - using remote clamd scanner ['ClamAV-clamdscan', 'clamdscan', "--stdout --no-summary --config-file=/etc/clamd-client.conf {}", [0], qr/:.*\sFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ], and you have a backup clamd scanner on a remote host. The /etc/clamd-client.conf is just a copy of your main /etc/clamd.conf, with changed: TCPSocket 3310 TCPAddr <remote-host-running-clamd> It would be more comfortable if clamdscan supported a command-line option to specify a host/port of a scanning host, but using an alternative config file works fine too for the time being. Someone should send a suggestion to ClamAV folks. ................++- Dunno. Perhaps. Because it says a copy, so clamav has still to be installed locally, then it will find and use it. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. wrote:
@av_scanners_backup = (
### http://www.clamav.net/ - backs up clamd or Mail::ClamAV ['ClamAV-clamscan', 'clamscan', "--stdout --no-summary -r --tempdir=$TEMPBASE {}", [0], qr/:.*\sFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) [FOUND$/m ],
That sounds a lot like the on-demand scan you were looking for?
Huh? Ah. That would be not using clamd, but clamav. It takes ages to start and load.
Yep. You can't have your cake and eat it. Anyway, you said latency and the extra work was not so important? This essentially describes the clamscan method: https://forum.iredmail.org/topic13607-a-solution-to-clamav-consuming-too-muc...
Found something. <https://sourceforge.net/p/amavis/mailman/message/23962953/>
[snip]
Dunno. Perhaps. Because it says a copy, so clamav has still to be installed locally, then it will find and use it.
Nah, remove the config, it isn't black magic. Besides, clamav doesn't need to be installed nor _running_ locally, you could just copy over the clamdscan binary. -- Per Jessen, Zürich (16.2°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/09/2019 13.52, Per Jessen wrote:
Carlos E. R. wrote:
@av_scanners_backup = (
### http://www.clamav.net/ - backs up clamd or Mail::ClamAV ['ClamAV-clamscan', 'clamscan', "--stdout --no-summary -r --tempdir=$TEMPBASE {}", [0], qr/:.*\sFOUND$/m, qr/^.*?: (?!Infected Archive)(.*) [FOUND$/m ],
That sounds a lot like the on-demand scan you were looking for?
Huh? Ah. That would be not using clamd, but clamav. It takes ages to start and load.
Yep. You can't have your cake and eat it. Anyway, you said latency and the extra work was not so important?
This essentially describes the clamscan method: https://forum.iredmail.org/topic13607-a-solution-to-clamav-consuming-too-muc...
No, that is not practical. As I said, this starts the scanner for every mail. One minute per mail. This moment I'm processing 1500 mails, that would be 25 hours. What I'd like is starting the clamd daemon on the first mail received, then it stays loaded till five minutes after the last mail is processed. *That* would be acceptable. It is also acceptable to have the process entirely in swap, restored when needed, which takes seconds, and then it goes back to swap a few minutes after the last use. Unfortunately, it is not possible to force a process to swap. A minute per mail, for every mail, is not acceptable.
Found something. <https://sourceforge.net/p/amavis/mailman/message/23962953/>
[snip]
Dunno. Perhaps. Because it says a copy, so clamav has still to be installed locally, then it will find and use it.
Nah, remove the config, it isn't black magic. Besides, clamav doesn't need to be installed nor _running_ locally, you could just copy over the clamdscan binary.
I'll have a better look. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
On 01/09/2019 16.01, Carlos E. R. wrote:
On 01/09/2019 14.51, Per Jessen wrote:
...
One second to process.
And, surprise! See memory use:
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND
3938 vscan 20 0 999648 121692 3832 624804 S 0,000 1,491 8:11.65 clamd
One second and a half CPU time used, less than a 100MB RES increase.
So you see, clamd can remain swapped out most of the time in my environment with no impairment.
Now, ten hours later, almost all is back on swap: top - 01:56:14 up 6 days, 14:07, 2 users, load average: 0,45, 0,95, 1,29 Tasks: 480 total, 1 running, 478 sleeping, 0 stopped, 1 zombie %Cpu(s): 3,0 us, 1,2 sy, 0,0 ni, 74,5 id, 21,3 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8161116 total, 716008 free, 3146676 used, 4298432 buff/cache KiB Swap: 25165820 total, 20471132 free, 4694688 used. 4542680 avail Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND ... 3938 vscan 20 0 1000020 28596 2600 721600 S 0,000 0,350 9:13.47 clamd 29786 vscan 20 0 193944 28524 4708 37008 S 0,000 0,350 0:01.02 /usr/sbin/amavi So the swapiness setting does work, but late. It is not aggressive. It has used a minute of CPU, though, probably freshclam - yes, an hour ago: <2.6> 2019-09-02 00:00:07 Telcontar postfix 4031 - - CB6C73213B5: removed <2.6> 2019-09-02 01:00:06 Telcontar clamd 3938 - - SelfCheck: Database status OK. <2.6> 2019-09-02 01:05:02 Telcontar freshclam 2293 - - Received signal: wake up <2.6> 2019-09-02 01:05:02 Telcontar freshclam 2293 - - ClamAV update process started at Mon Sep 2 01:05:02 2019 <2.4> 2019-09-02 01:05:02 Telcontar freshclam 2293 - - Your ClamAV installation is OUTDATED! <2.4> 2019-09-02 01:05:02 Telcontar freshclam 2293 - - Local version: 0.101.3 Recommended version: 0.101.4 <2.6> 2019-09-02 01:05:02 Telcontar freshclam 2293 - - DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav <2.6> 2019-09-02 01:05:02 Telcontar freshclam 2293 - - main.cld is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr) <2.6> 2019-09-02 01:05:02 Telcontar freshclam 2293 - - daily.cld is up to date (version: 25559, sigs: 1745720, f-level: 63, builder: raynman) <2.6> 2019-09-02 01:05:02 Telcontar freshclam 2293 - - bytecode.cld is up to date (version: 330, sigs: 94, f-level: 63, builder: neo) <2.6> 2019-09-02 01:05:02 Telcontar freshclam 2293 - - -------------------------------------- <2.6> 2019-09-02 01:52:41 Telcontar postfix 5302 - - 1521B3213B4: uid=0 from=<root> -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
On 31/08/2019 22.21, Carlos E. R. wrote:
Photo using top, sort by memory, this instant:
PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21395 cer 20 0 3402620 782716 85432 166280 S 69,35 9,591 61:58.83 Web Content 1096 cer 20 0 3688736 767644 104104 0 S 2,976 9,406 19:05.04 thunderbird-bin 3938 vscan 20 0 999648 725460 2628 17856 S 0,000 8,889 8:09.22 clamd .................................*******
0.7 GB of RAM! I will have to uninstall it. Las time I looked, it was half a gigabyte.
And it is set with swapiness of 100...
I read this, googling: ]> cgroups may work for you. You can apply memory limit and once ]> the process hits that limit, it will begin to swap. I'm already using cgroups. I have swapiness set to 100 for its cgroup. Telcontar:~ # cat /etc/systemd/system/clamd.service.d/override.conf [Service] # Modifies /usr/lib/systemd/system/clamd.service so that clamd runs with a cgroup that increases swappiness to 100 # thus reducing wasted memory when it is not actively running. Although swapping happens slowly, hours. ExecStartPre=/bin/sh -c "mkdir /sys/fs/cgroup/memory/clamd" ExecStartPre=/bin/sh -c "echo 100 > /sys/fs/cgroup/memory/clamd/memory.swappiness" ExecStartPost=/bin/sh -c "echo $MAINPID > /sys/fs/cgroup/memory/clamd/cgroup.procs" ExecStopPost=/bin/sh -c "rmdir /sys/fs/cgroup/memory/clamd" # ControlGroupAttribute=memory.swappiness 100 #The option was removed: #https://bugzilla.redhat.com/show_bug.cgi?id=1172890 # https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/htm... # http://0pointer.de/blog/projects/resources.html # Managing Other Resource Parameters # https://www.freedesktop.org/software/systemd/man/systemd.unit.html Telcontar:~ # Now, what memory limit can I use so that if clamd uses more than that, it swaps? It seems to be data store. ]> On Linux, cgroups can have a memory limit beyond which processes in that group page out. ]> A systemd service can set such a limit with the MemoryLimit directive, see the man page ]> or RHEL's resource management guide. Remember that systemd units can be per user, root privilege not required. It is this (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...): [Service] MemoryLimit=1G I set 200MB, and indeed, it swaps early, impacting startup heavily. So heavily that it does not start: Telcontar:~ # systemctl restart clamd Job for clamd.service failed because a timeout was exceeded. See "systemctl status clamd.service" and "journalctl -xe" for details. Telcontar:~ # This is not what I want... I want the limit to be applied after a delay. Even with: MemoryLimit=500M it fails to start, and trashes during start, which is not what I want. I can make do, while I study moving to another machine: MemoryLimit=500M TimeoutSec=300s Result after start succeeds is: top - 14:44:20 up 2:31, 2 users, load average: 0,52, 0,56, 0,57 Tasks: 452 total, 1 running, 450 sleeping, 0 stopped, 1 zombie %Cpu(s): 10,0 us, 1,1 sy, 0,1 ni, 87,3 id, 1,5 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 8161116 total, 639140 free, 5095308 used, 2426668 buff/cache KiB Swap: 25165820 total, 24423676 free, 742144 used. 2576044 avail Mem PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND ... 30239 vscan 20 0 923168 2092 2092 728904 S 0,000 0,026 0:00.00 clamd Which is indeed what I wanted! Bingo! -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. wrote:
MemoryLimit=500M TimeoutSec=300s
Result after start succeeds is:
[snip]
Which is indeed what I wanted! Bingo!
Interesting experiment. How long does it take to scan an email now? I tried the same on my own clamd daemon (custom written, but same idea). Startup took just about 9 minutes, the first few scans took up to 35sec, but then it seems to drop to 1-2 seconds. That's easily 10 times more than normal, which isn't good. Sometimes processing time shoots up to 5 or 10. Still fairly minimal impact - I guess the numbers I'm seeing might be schewed as the system has plenty of memory. -- Per Jessen, Zürich (16.8°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/09/2019 16.03, Per Jessen wrote:
Carlos E. R. wrote:
MemoryLimit=500M TimeoutSec=300s
Result after start succeeds is:
[snip]
Which is indeed what I wanted! Bingo!
Interesting experiment. How long does it take to scan an email now?
One with virii, instantly: <2.6> 2019-09-02 16:25:52 Telcontar postfix 4022 - - E4CD6320B4A: from=<cer@Isengard.valinor>, size=408087, nrcpt=1 (queue active) <2.6> 2019-09-02 16:25:52 Telcontar postfix 3162 - - disconnect from Isengard.valinor[192.168.1.16] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 <2.6> 2019-09-02 16:25:52 Telcontar amavis 3937 - - (03937-05) LMTP [127.0.0.1]:10024 /var/spool/amavis/tmp/amavis-20190902T130130-03937-Cu9e3pu9: <cer@Isengard.valinor> -> <cer@Telcontar.valinor> SIZE=408087 Received: from Telcontar.valinor ([127.0.0.1]) by localhost (telcontar.valinor [127.0.0.1]) (amavisd-new, port 10024) with LMTP for <cer@telcontar.valinor>; Mon, 2 Sep 2019 16:25:52 +0200 (CEST) <2.6> 2019-09-02 16:25:53 Telcontar amavis 3937 - - (03937-05) Checking: ddPNTIuWd5YP MYNETS [192.168.1.16] <cer@Isengard.valinor> -> <cer@Telcontar.valinor> <2.6> 2019-09-02 16:25:53 Telcontar amavis 3937 - - (03937-05) p.path BANNED:1 cer@Telcontar.valinor: "P=p003,L=1,M=multipart/mixed | P=p002,L=1/2,M=application/octet-stream,T=exe,T=exe-ms,N=virisample", matching_key="(?^:^\\.(exe-ms|dll)$)" <2.6> 2019-09-02 16:25:55 Telcontar clamd 30239 - - /var/spool/amavis/tmp/amavis-20190902T130130-03937-Cu9e3pu9/parts/p004: Win.Worm.N-74 FOUND <2.6> 2019-09-02 16:25:55 Telcontar clamd 30239 - - /var/spool/amavis/tmp/amavis-20190902T130130-03937-Cu9e3pu9/parts/p002: Win.Worm.N-74 FOUND <2.6> 2019-09-02 16:25:55 Telcontar amavis 3937 - - (03937-05) local delivery: <> -> virus-quarantine, mbx=/var/spool/amavis/virusmails/virus-ddPNTIuWd5YP <2.6> 2019-09-02 16:25:55 Telcontar postfix 3173 - - connect from localhost[127.0.0.1] I'll try now a healthy windows executable. [...] yes, it takes much longer: 2.6> 2019-09-02 16:30:56 Telcontar postfix 3437 - - connect from Isengard.valinor[192.168.1.16] <2.6> 2019-09-02 16:30:56 Telcontar postfix 3437 - - 97451320B4A: client=Isengard.valinor[192.168.1.16] <2.6> 2019-09-02 16:30:56 Telcontar postfix 3441 - - 97451320B4A: message-id=<alpine.LSU.2.21.1909021630300.11050@Isengard.valinor> <2.6> 2019-09-02 16:30:56 Telcontar postfix 4022 - - 97451320B4A: from=<cer@Isengard.valinor>, size=21665824, nrcpt=1 (queue active) <2.6> 2019-09-02 16:30:56 Telcontar postfix 3437 - - disconnect from Isengard.valinor[192.168.1.16] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 <2.6> 2019-09-02 16:30:57 Telcontar amavis 3939 - - (03939-05) LMTP [127.0.0.1]:10024 /var/spool/amavis/tmp/amavis-20190902T130850-03939-Cgk2L9yY: <cer@Isengard.valinor> -> <cer@Telcontar.valinor> SIZE=21665824 Received: from Telcontar.valinor ([127.0.0.1]) by localhost (telcontar.valinor [127.0.0.1]) (amavisd-new, port 10024) with LMTP for <cer@telcontar.valinor>; Mon, 2 Sep 2019 16:30:57 +0200 (CEST) <2.6> 2019-09-02 16:30:57 Telcontar amavis 3939 - - (03939-05) Checking: J0be6HvIx64w MYNETS [192.168.1.16] <cer@Isengard.valinor> -> <cer@Telcontar.valinor> <2.6> 2019-09-02 16:30:57 Telcontar amavis 3939 - - (03939-05) p.path BANNED:1 cer@Telcontar.valinor: "P=p003,L=1,M=multipart/mixed | P=p002,L=1/2,M=application/x-executable,T=exe,T=exe-ms,N=Restorer7.exe", matching_key="(?^:^\\.(exe-ms|dll)$)" <2.6> 2019-09-02 16:31:46 Telcontar dovecot - - - imap(cer)<3323><CNLOy5KRZLh/AAAB>: Logged out in=3973 out=10314 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0 <2.6> 2019-09-02 16:32:16 Telcontar dovecot - - - imap-login: Login: user=<cer>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=3498, TLS, session=<RFyq1pKRerh/AAAB> <2.6> 2019-09-02 16:32:29 Telcontar amavis 3939 - - (03939-05) local delivery: <> -> banned-quarantine, mbx=/var/spool/amavis/virusmails/banned-J0be6HvIx64w Took two minutes and a half, I saw clamd busy and trashing. But I do not see a clamd entry in the log, maybe because it is not a virus. <2.6> 2019-09-02 16:32:30 Telcontar postfix 3503 - - connect from localhost[127.0.0.1] <2.6> 2019-09-02 16:32:30 Telcontar postfix 3503 - - 0B4303213B5: client=localhost[127.0.0.1] <2.6> 2019-09-02 16:32:30 Telcontar postfix 3441 - - 0B4303213B5: message-id=<VAJ0be6HvIx64w@telcontar.valinor> <2.6> 2019-09-02 16:32:30 Telcontar postfix 3503 - - disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 <2.6> 2019-09-02 16:32:30 Telcontar postfix 4022 - - 0B4303213B5: from=<virusalert@telcontar.valinor>, size=2917, nrcpt=1 (queue active) <2.6> 2019-09-02 16:32:30 Telcontar amavis 3939 - - (03939-05) Gru4ztQtvhFO(J0be6HvIx64w) SEND from <virusalert@telcontar.valinor> -> <virusalert@telcontar.valinor>, ENVID=AM.Gru4ztQtvhFO.20190902T143230Z@telcontar.valinor 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 0B4303213B5 <2.6> 2019-09-02 16:32:30 Telcontar amavis 3939 - - (03939-05) DEFANGING MAIL: WARNING: banning rules detected suspect part(s),\ndo not open unless you know what you are doing <2.6> 2019-09-02 16:32:30 Telcontar postfix 3503 - - connect from localhost[127.0.0.1] And then a pdf: 2.6> 2019-09-02 16:34:13 Telcontar postfix 3577 - - connect from Isengard.valinor[192.168.1.16] <2.6> 2019-09-02 16:34:13 Telcontar postfix 3577 - - AD196320B4A: client=Isengard.valinor[192.168.1.16] <2.6> 2019-09-02 16:34:13 Telcontar postfix 3578 - - AD196320B4A: message-id=<alpine.LSU.2.21.1909021633580.11149@Isengard.valinor> <2.6> 2019-09-02 16:34:13 Telcontar postfix 4022 - - AD196320B4A: from=<cer@Isengard.valinor>, size=3115, nrcpt=1 (queue active) <2.6> 2019-09-02 16:34:13 Telcontar postfix 3577 - - disconnect from Isengard.valinor[192.168.1.16] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 <2.6> 2019-09-02 16:34:13 Telcontar amavis 3937 - - (03937-06) LMTP [127.0.0.1]:10024 /var/spool/amavis/tmp/amavis-20190902T130130-03937-Cu9e3pu9: <cer@Isengard.valinor> -> <cer@Telcontar.valinor> SIZE=3115 Received: from Telcontar.valinor ([127.0.0.1]) by localhost (telcontar.valinor [127.0.0.1]) (amavisd-new, port 10024) with LMTP for <cer@telcontar.valinor>; Mon, 2 Sep 2019 16:34:13 +0200 (CEST) <2.6> 2019-09-02 16:34:13 Telcontar amavis 3937 - - (03937-06) Checking: u_8kSrBam1lD MYNETS [192.168.1.16] <cer@Isengard.valinor> -> <cer@Telcontar.valinor> <2.6> 2019-09-02 16:34:13 Telcontar postfix 3582 - - connect from localhost[127.0.0.1] Not even a second. See the mem progression: 3929 vscan 20 0 997064 738932 4624 0 S 0,000 9,054 0:00.10 clamd 30239 vscan 20 0 923168 2092 2092 728904 S 0,000 0,026 0:00.00 clamd 3929 vscan 20 0 997064 738932 4624 0 S 0,000 9,054 0:00.10 clamd 30239 vscan 20 0 923168 2092 2092 728904 S 0,000 0,026 0:00.00 clamd 30239 vscan 20 0 996900 30004 4556 704460 S 0,000 0,368 0:00.22 clamd 30239 vscan 20 0 996900 29204 4620 704460 S 0,000 0,358 0:00.22 clamd 30239 vscan 20 0 996900 32652 4716 701284 S 0,000 0,400 0:00.29 clamd 30239 vscan 20 0 996900 123228 4808 615560 S 0,000 1,510 0:01.49 clamd 30239 vscan 20 0 996900 30004 4556 704460 S 0,000 0,368 0:00.22 clamd 30239 vscan 20 0 996900 29204 4620 704460 S 0,000 0,358 0:00.22 clamd 30239 vscan 20 0 996900 32652 4716 701284 S 0,000 0,400 0:00.29 clamd 30239 vscan 20 0 996900 123228 4808 615560 S 0,000 1,510 0:01.49 clamd - virus detected. 30239 vscan 20 0 998992 360616 6756 424364 S 0,000 4,419 1:31.16 clamd - healthy exe 30239 vscan 20 0 998992 361200 6780 423740 S 0,000 4,426 1:31.18 clamd - pdf So there is a long delay only when there is an executable that is not detected as virus, because then it has to load the entire data section. Notice however that amavis rejects it nonetheless, I have it setup that way.
I tried the same on my own clamd daemon (custom written, but same idea). Startup took just about 9 minutes, the first few scans took up to 35sec, but then it seems to drop to 1-2 seconds. That's easily 10 times more than normal, which isn't good. Sometimes processing time shoots up to 5 or 10. Still fairly minimal impact - I guess the numbers I'm seeing might be schewed as the system has plenty of memory.
Yes, I can accept this impact. Now ram usage is just beneath 400M. I'll try later something: find out the cgroup entry for memory limit and change it "brute", ie, after start. See if I can manage to drop the mem use to minimal on demand, then back to "normal" limit. This would allow a much faster response, and recover 800MB when I want it. If this works, I may be able to script it into the mail fetch script. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. wrote:
I tried the same on my own clamd daemon (custom written, but same idea). Startup took just about 9 minutes, the first few scans took up to 35sec, but then it seems to drop to 1-2 seconds. That's easily 10 times more than normal, which isn't good. Sometimes processing time shoots up to 5 or 10. Still fairly minimal impact - I guess the numbers I'm seeing might be schewed as the system has plenty of memory.
Yes, I can accept this impact. Now ram usage is just beneath 400M. I'll try later something: find out the cgroup entry for memory limit and change it "brute", ie, after start.
memory.limit_in_bytes
See if I can manage to drop the mem use to minimal on demand, then back to "normal" limit. This would allow a much faster response, and recover 800MB when I want it. If this works, I may be able to script it into the mail fetch script.
I think you can probably find a fairly low "working-set" that'll give you a decent response in 99% of cases. However, the reason you're not seeing your clamd reduce memory usage without this memory limit is that your system simply has sufficient memory. I took a look at my test system cluster, four virtual machines with only 1.2Gb each. On those clamd is using less than 500Mb, with no cgroup memory limit and no changed swappiness. -- Per Jessen, Zürich (16.8°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/09/2019 17.36, Per Jessen wrote:
Carlos E. R. wrote:
I tried the same on my own clamd daemon (custom written, but same idea). Startup took just about 9 minutes, the first few scans took up to 35sec, but then it seems to drop to 1-2 seconds. That's easily 10 times more than normal, which isn't good. Sometimes processing time shoots up to 5 or 10. Still fairly minimal impact - I guess the numbers I'm seeing might be schewed as the system has plenty of memory.
Yes, I can accept this impact. Now ram usage is just beneath 400M. I'll try later something: find out the cgroup entry for memory limit and change it "brute", ie, after start.
memory.limit_in_bytes
See if I can manage to drop the mem use to minimal on demand, then back to "normal" limit. This would allow a much faster response, and recover 800MB when I want it. If this works, I may be able to script it into the mail fetch script.
I think you can probably find a fairly low "working-set" that'll give you a decent response in 99% of cases.
I'm considering alternatives, like hacking amavis so that it "unswaps" clamav before calling it, then sets the limit to 200 afterwards. Or doing it manually. Or a cronjob: if clamav is not busy (below 1%) force it to swap, then rever after a minute (but it will stay swapped out, I hope).
However, the reason you're not seeing your clamd reduce memory usage without this memory limit is that your system simply has sufficient memory. I took a look at my test system cluster, four virtual machines with only 1.2Gb each. On those clamd is using less than 500Mb, with no cgroup memory limit and no changed swappiness.
No, this is not so. Several applications, like Thunderbird and Mozilla, are starving and they swap. I prefer clamav to go to swap than others, simple as that. Yesterday night the system had more than four gigabytes in swap. On hibernation, it failed to hibernate. Then it failed to init 3, did nothing. On poweroff, it stalled, and finally I had to REISUB. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. wrote:
On 02/09/2019 17.36, Per Jessen wrote:
Carlos E. R. wrote:
I tried the same on my own clamd daemon (custom written, but same idea). Startup took just about 9 minutes, the first few scans took up to 35sec, but then it seems to drop to 1-2 seconds. That's easily 10 times more than normal, which isn't good. Sometimes processing time shoots up to 5 or 10. Still fairly minimal impact - I guess the numbers I'm seeing might be schewed as the system has plenty of memory.
Yes, I can accept this impact. Now ram usage is just beneath 400M. I'll try later something: find out the cgroup entry for memory limit and change it "brute", ie, after start.
memory.limit_in_bytes
See if I can manage to drop the mem use to minimal on demand, then back to "normal" limit. This would allow a much faster response, and recover 800MB when I want it. If this works, I may be able to script it into the mail fetch script.
I think you can probably find a fairly low "working-set" that'll give you a decent response in 99% of cases.
I'm considering alternatives, like hacking amavis so that it "unswaps" clamav before calling it, then sets the limit to 200 afterwards. Or doing it manually. Or a cronjob: if clamav is not busy (below 1%) force it to swap, then rever after a minute (but it will stay swapped out, I hope).
Sounds a bit over-engineered.
However, the reason you're not seeing your clamd reduce memory usage without this memory limit is that your system simply has sufficient memory. I took a look at my test system cluster, four virtual machines with only 1.2Gb each. On those clamd is using less than 500Mb, with no cgroup memory limit and no changed swappiness.
No, this is not so.
In my example, clamd seems to behave the way you want though.
Several applications, like Thunderbird and Mozilla, are starving and they swap. I prefer clamav to go to swap than others, simple as that.
There is probably a way of prioritising that. I wonder if processes running with privileged userids are given higher priority? You could try running clamd with a non-privileged userid. -- Per Jessen, Zürich (15.5°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/09/2019 17.36, Per Jessen wrote:
Carlos E. R. wrote:
I tried the same on my own clamd daemon (custom written, but same idea). Startup took just about 9 minutes, the first few scans took up to 35sec, but then it seems to drop to 1-2 seconds. That's easily 10 times more than normal, which isn't good. Sometimes processing time shoots up to 5 or 10. Still fairly minimal impact - I guess the numbers I'm seeing might be schewed as the system has plenty of memory.
Yes, I can accept this impact. Now ram usage is just beneath 400M. I'll try later something: find out the cgroup entry for memory limit and change it "brute", ie, after start.
memory.limit_in_bytes
Does not seem it: Telcontar:~ # cat /sys/fs/cgroup/memory/clamd/memory.limit_in_bytes 9223372036854771712 Telcontar:~ # This is more likely, the value is similar to what I wrote in the cgroup config (500M): cer@Telcontar:~> cat /sys/fs/cgroup/memory/clamd/memory.max_usage_in_bytes 496934912 <===== cer@Telcontar:~> cat /sys/fs/cgroup/memory/clamd/memory.usage_in_bytes 172032 cer@Telcontar:~> cat /sys/fs/cgroup/memory/clamd/memory.stat Because: cer@Telcontar:~> cat /etc/systemd/system/clamd.service.d/override.conf | grep -i memorylimit MemoryLimit=500M cer@Telcontar:~> If I change it: Telcontar:~ # echo 700000000 > /sys/fs/cgroup/memory/clamd/memory.max_usage_in_bytes Telcontar:~ # cat /sys/fs/cgroup/memory/clamd/memory.max_usage_in_bytes 172032 Telcontar:~ # echo 700000000 > /sys/fs/cgroup/memory/clamd/memory.max_usage_in_bytes Telcontar:~ # cat /sys/fs/cgroup/memory/clamd/memory.max_usage_in_bytes 172032 Telcontar:~ # Doesn't seem it will work. The process was and is entirely swapped out, fresh out of hibernation - not even code: PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 30239 vscan 20 0 998992 0 0 778308 S 0,000 0,000 1:31.32 clamd Other possibilities: memory.kmem.limit_in_bytes 9223372036854771712 memory.kmem.max_usage_in_bytes 520192 memory.kmem.usage_in_bytes 155648 memory.limit_in_bytes 9223372036854771712 memory.max_usage_in_bytes 172032 memory.soft_limit_in_bytes 9223372036854771712 memory.usage_in_bytes 172032 -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. wrote:
On 02/09/2019 17.36, Per Jessen wrote:
Carlos E. R. wrote:
I tried the same on my own clamd daemon (custom written, but same idea). Startup took just about 9 minutes, the first few scans took up to 35sec, but then it seems to drop to 1-2 seconds. That's easily 10 times more than normal, which isn't good. Sometimes processing time shoots up to 5 or 10. Still fairly minimal impact - I guess the numbers I'm seeing might be schewed as the system has plenty of memory.
Yes, I can accept this impact. Now ram usage is just beneath 400M. I'll try later something: find out the cgroup entry for memory limit and change it "brute", ie, after start.
memory.limit_in_bytes
Does not seem it:
Telcontar:~ # cat /sys/fs/cgroup/memory/clamd/memory.limit_in_bytes 9223372036854771712
That is odd. On my 15.1 system, the complete path is: /sys/fs/cgroup/memory/system.slice/bwclamd.service/memory.limit_in_bytes (bwclamd is our custom clamd). I used "memory.limit_in_bytes", and it works. -- Per Jessen, Zürich (20.1°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/09/2019 14.40, Per Jessen wrote:
Carlos E. R. wrote:
On 02/09/2019 17.36, Per Jessen wrote:
Carlos E. R. wrote:
I tried the same on my own clamd daemon (custom written, but same idea). Startup took just about 9 minutes, the first few scans took up to 35sec, but then it seems to drop to 1-2 seconds. That's easily 10 times more than normal, which isn't good. Sometimes processing time shoots up to 5 or 10. Still fairly minimal impact - I guess the numbers I'm seeing might be schewed as the system has plenty of memory.
Yes, I can accept this impact. Now ram usage is just beneath 400M. I'll try later something: find out the cgroup entry for memory limit and change it "brute", ie, after start.
memory.limit_in_bytes
Does not seem it:
Telcontar:~ # cat /sys/fs/cgroup/memory/clamd/memory.limit_in_bytes 9223372036854771712
That is odd. On my 15.1 system, the complete path is:
/sys/fs/cgroup/memory/system.slice/bwclamd.service/memory.limit_in_bytes
Ah! "/sys/fs/cgroup/memory/system.slice/clamd.service/" also exist here. I did not know about it. My service file have these additions: ExecStartPre=/bin/sh -c "mkdir /sys/fs/cgroup/memory/clamd" ExecStartPre=/bin/sh -c "echo 100 > /sys/fs/cgroup/memory/clamd/memory.swappiness" ExecStartPost=/bin/sh -c "echo $MAINPID > /sys/fs/cgroup/memory/clamd/cgroup.procs" ExecStopPost=/bin/sh -c "rmdir /sys/fs/cgroup/memory/clamd" I wrote those, I did not know about the other place. That may be the reason that swapiness=100 is not working.
(bwclamd is our custom clamd).
I used "memory.limit_in_bytes", and it works.
I will have to change my service, obviously. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
On 03/09/2019 15.04, Carlos E. R. wrote:
On 03/09/2019 14.40, Per Jessen wrote:
Carlos E. R. wrote:
That is odd. On my 15.1 system, the complete path is:
/sys/fs/cgroup/memory/system.slice/bwclamd.service/memory.limit_in_bytes
Ah!
"/sys/fs/cgroup/memory/system.slice/clamd.service/" also exist here. I did not know about it.
My service file have these additions:
ExecStartPre=/bin/sh -c "mkdir /sys/fs/cgroup/memory/clamd" ExecStartPre=/bin/sh -c "echo 100 > /sys/fs/cgroup/memory/clamd/memory.swappiness" ExecStartPost=/bin/sh -c "echo $MAINPID > /sys/fs/cgroup/memory/clamd/cgroup.procs"
ExecStopPost=/bin/sh -c "rmdir /sys/fs/cgroup/memory/clamd"
I wrote those, I did not know about the other place. That may be the reason that swapiness=100 is not working.
(bwclamd is our custom clamd).
I used "memory.limit_in_bytes", and it works.
I will have to change my service, obviously.
Problem: [Service] ExecStartPost=/bin/sh -c "echo 100 > /sys/fs/cgroup/memory/system.slice/clamd.service/memory.swappiness" <3.6> 2019-09-03 15:21:27 Telcontar systemd 1 - - Starting Clamav antivirus Deamon... <3.6> 2019-09-03 15:22:25 Telcontar sh 18988 - - /bin/sh: /sys/fs/cgroup/memory/system.slice/clamd.service/memory.swappiness: No such file or directory <3.5> 2019-09-03 15:22:25 Telcontar systemd 1 - - clamd.service: Control process exited, code=exited status=1 <3.3> 2019-09-03 15:22:26 Telcontar systemd 1 - - Failed to start Clamav antivirus Deamon. <3.5> 2019-09-03 15:22:26 Telcontar systemd 1 - - clamd.service: Unit entered failed state. <3.4> 2019-09-03 15:22:26 Telcontar systemd 1 - - clamd.service: Failed with result 'exit-code'. I will have to remove my entry and find out which is the actual directory. [...] None. The systemd cgroup directory does not exist for clamd: Telcontar:~ # ls /sys/fs/cgroup/memory/system.slice/cl* ls: cannot access '/sys/fs/cgroup/memory/system.slice/cl*': No such file or directory Telcontar:~ # Telcontar:~ # ls /sys/fs/cgroup/memory/system.slice/ | grep -i clam freshclam.service Telcontar:~ # How do I make it be created? -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
On 03/09/2019 15.47, Carlos E. R. wrote:
On 03/09/2019 15.04, Carlos E. R. wrote:
I will have to remove my entry and find out which is the actual directory. [...]
None. The systemd cgroup directory does not exist for clamd:
Telcontar:~ # ls /sys/fs/cgroup/memory/system.slice/cl* ls: cannot access '/sys/fs/cgroup/memory/system.slice/cl*': No such file or directory Telcontar:~ #
Telcontar:~ # ls /sys/fs/cgroup/memory/system.slice/ | grep -i clam freshclam.service Telcontar:~ #
How do I make it be created?
I don't understand. I see several services with cgroups, like freshclam, spamd, amavis... just not clamd (and yes, the service is running). I see it outside of the tree: Telcontar:~ # cat /sys/fs/cgroup/memory/system.slice/tasks 19248 19249 Telcontar:~ # top -b -n 1 | grep 19248 19248 vscan 20 0 996900 726536 4760 S 0.000 8.902 0:00.08 clamd Telcontar:~ # top -b -n 1 | grep 19249 Telcontar:~ # Why? I do not want it there. I can't adjust it there. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. wrote:
On 03/09/2019 15.47, Carlos E. R. wrote:
On 03/09/2019 15.04, Carlos E. R. wrote:
I will have to remove my entry and find out which is the actual directory. [...]
None. The systemd cgroup directory does not exist for clamd:
Telcontar:~ # ls /sys/fs/cgroup/memory/system.slice/cl* ls: cannot access '/sys/fs/cgroup/memory/system.slice/cl*': No such file or directory Telcontar:~ #
Telcontar:~ # ls /sys/fs/cgroup/memory/system.slice/ | grep -i clam freshclam.service Telcontar:~ #
How do I make it be created?
Create /etc/systemd/systemd/clamd.service.d/extra.conf and add "MemoryLimit=500M" to that file. -- Per Jessen, Zürich (20.5°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/09/2019 16.08, Per Jessen wrote:
Carlos E. R. wrote:
Telcontar:~ # ls /sys/fs/cgroup/memory/system.slice/ | grep -i clam freshclam.service Telcontar:~ #
How do I make it be created?
Create /etc/systemd/systemd/clamd.service.d/extra.conf
and add "MemoryLimit=500M" to that file.
Yes, I just did that. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
On 03/09/2019 16.01, Carlos E. R. wrote:
On 03/09/2019 15.47, Carlos E. R. wrote:
On 03/09/2019 15.04, Carlos E. R. wrote:
I will have to remove my entry and find out which is the actual directory. [...]
None. The systemd cgroup directory does not exist for clamd:
Telcontar:~ # ls /sys/fs/cgroup/memory/system.slice/cl* ls: cannot access '/sys/fs/cgroup/memory/system.slice/cl*': No such file or directory Telcontar:~ #
Telcontar:~ # ls /sys/fs/cgroup/memory/system.slice/ | grep -i clam freshclam.service Telcontar:~ #
How do I make it be created?
I don't understand. I see several services with cgroups, like freshclam, spamd, amavis... just not clamd (and yes, the service is running).
I see it outside of the tree:
Telcontar:~ # cat /sys/fs/cgroup/memory/system.slice/tasks 19248 19249
Telcontar:~ # top -b -n 1 | grep 19248 19248 vscan 20 0 996900 726536 4760 S 0.000 8.902 0:00.08 clamd Telcontar:~ # top -b -n 1 | grep 19249 Telcontar:~ #
Why? I do not want it there. I can't adjust it there.
I'll activate: MemoryLimit=700M to see if the cgroup is created. [...] Yes, now I have it. /sys/fs/cgroup/memory/system.slice/clamd.service/ [...] Editing: [Service] ExecStartPost=/bin/sh -c "echo 100 > /sys/fs/cgroup/memory/system.slice/clamd.service/memory.swappiness" MemoryLimit=700M TimeoutSec=300s And it works: PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21500 vscan 20 0 923168 3896 2008 727204 S 0,000 0,048 0:00.00 clamd Just after starting it: Telcontar:~ # systemctl status clamd.service ● clamd.service - Clamav antivirus Deamon Loaded: loaded (/usr/lib/systemd/system/clamd.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system/clamd.service.d └─override.conf Active: active (running) since Tue 2019-09-03 16:06:32 CEST; 2min 13s ago Main PID: 21500 (clamd) Tasks: 2 (limit: 4915) Memory: 209.2M (limit: 700.0M) <============ CGroup: /system.slice/clamd.service └─21500 /usr/sbin/clamd Sep 03 16:06:18 Telcontar clamd[21500]: ELF support enabled. Sep 03 16:06:18 Telcontar clamd[21500]: Mail files support enabled. Sep 03 16:06:18 Telcontar clamd[21500]: OLE2 support enabled. Sep 03 16:06:18 Telcontar clamd[21500]: PDF support enabled. Sep 03 16:06:18 Telcontar clamd[21500]: SWF support enabled. Sep 03 16:06:18 Telcontar clamd[21500]: HTML support enabled. Sep 03 16:06:18 Telcontar clamd[21500]: XMLDOCS support enabled. Sep 03 16:06:18 Telcontar clamd[21500]: HWP3 support enabled. Sep 03 16:06:18 Telcontar clamd[21500]: Self checking every 3600 seconds. Sep 03 16:06:32 Telcontar systemd[1]: Started Clamav antivirus Deamon. Telcontar:~ # It takes minutes to start it, one core 100% <3.6> 2019-09-03 16:02:40 Telcontar systemd 1 - - Starting Clamav antivirus Deamon... <3.6> 2019-09-03 16:03:51 Telcontar systemd 1 - - Started Clamav antivirus Deamon. <3.6> 2019-09-03 16:05:09 Telcontar systemd 1 - - Reloading. <3.4> 2019-09-03 16:05:10 Telcontar systemd 1 - - nss-lookup.target: Dependency Before=nss-lookup.target dropped <3.6> 2019-09-03 16:05:13 Telcontar systemd 1 - - Stopping Clamav antivirus Deamon... <3.6> 2019-09-03 16:05:21 Telcontar systemd 1 - - Stopped Clamav antivirus Deamon. <3.6> 2019-09-03 16:05:21 Telcontar systemd 1 - - Starting Clamav antivirus Deamon... <3.6> 2019-09-03 16:06:32 Telcontar systemd 1 - - Started Clamav antivirus Deamon. <3.6> 2019-09-03 16:07:42 Telcontar systemd 1 - - Reloading. Now I will email me a clean executable, the worst case: <2.6> 2019-09-03 16:12:55 Telcontar postfix 21848 - - disconnect from Isengard.valinor[192.168.1.16] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 <2.6> 2019-09-03 16:12:55 Telcontar amavis 3937 - - (03937-17) LMTP [127.0.0.1]:10024 /var/spool/amavis/tmp/amavis-20190902T130130-03937-Cu9e3pu9: <cer@Isengard.valinor> -> <cer@Telcontar.valinor> SIZE=21665944 Received: from Telcontar.valinor ([127.0.0.1]) by localhost (telcontar.valinor [127.0.0.1]) (amavisd-new, port 10024) with LMTP for <cer@telcontar.valinor>; Tue, 3 Sep 2019 16:12:55 +0200 (CEST) <2.6> 2019-09-03 16:12:55 Telcontar amavis 3937 - - (03937-17) Checking: Xq36SfjtRKYY MYNETS [192.168.1.16] <cer@Isengard.valinor> -> <cer@Telcontar.valinor> <2.6> 2019-09-03 16:12:55 Telcontar amavis 3937 - - (03937-17) p.path BANNED:1 cer@Telcontar.valinor: "P=p003,L=1,M=multipart/mixed | P=p002,L=1/2,M=application/x-executable,T=exe,T=exe-ms,N=Restorer7.exe", matching_key="(?^:^\\.(exe-ms|dll)$)" <2.6> 2019-09-03 16:14:33 Telcontar amavis 3937 - - (03937-17) local delivery: <> -> banned-quarantine, mbx=/var/spool/amavis/virusmails/banned-Xq36SfjtRKYY <2.6> 2019-09-03 16:14:33 Telcontar postfix 21920 - - connect from localhost[127.0.0.1] About a minute and a half, that's acceptable. PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21500 vscan 20 0 998992 360596 6712 424492 S 0,000 4,418 1:31.42 clamd Now it is not fully swaped out. Let's try forcing it: Telcontar:~ # cat /sys/fs/cgroup/memory/system.slice/clamd.service/memory.limit_in_bytes 734003200 Telcontar:~ # echo 200000000 > /sys/fs/cgroup/memory/system.slice/clamd.service/memory.limit_in_bytes Telcontar:~ # cat /sys/fs/cgroup/memory/system.slice/clamd.service/memory.limit_in_bytes 199999488 Telcontar:~ # PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21500 vscan 20 0 998992 180672 6596 604300 S 0,000 2,214 1:31.42 clamd Telcontar:~ # echo 700000000 > /sys/fs/cgroup/memory/system.slice/clamd.service/memory.limit_in_bytes Telcontar:~ # cat /sys/fs/cgroup/memory/system.slice/clamd.service/memory.limit_in_bytes 699998208 Telcontar:~ # PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21500 vscan 20 0 998992 180672 6596 604300 S 0,000 2,214 1:31.42 clamd This is very nice, I can easily script that to force clamd to swap :-)) Telcontar:~ # echo 50000000 > /sys/fs/cgroup/memory/system.slice/clamd.service/memory.limit_in_bytes [...] Telcontar:~ # echo 700000000 > /sys/fs/cgroup/memory/system.slice/clamd.service/memory.limit_in_bytes PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21500 vscan 20 0 998992 53124 6596 731848 S 0,000 0,651 1:31.42 clamd *very* nice. :-))) Let's try sending a pdf. [...] instant. clamav was active briefly. PID USER PR NI VIRT RES SHR SWAP S %CPU %MEM TIME+ COMMAND 21500 vscan 20 0 998992 71768 6756 713376 S 0,000 0,879 1:31.65 clamd 2.6> 2019-09-03 16:24:44 Telcontar postfix 4022 - - 7DD473213B7: from=<cer@Isengard.valinor>, size=3805, nrcpt=1 (queue active) <2.6> 2019-09-03 16:24:44 Telcontar amavis 3939 - - (03939-17) v-4c8WQA_QSF FWD from <cer@Isengard.valinor> -> <cer@Telcontar.valinor>, BODY=7BIT 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7DD473213B7 <2.5> 2019-09-03 16:24:44 Telcontar amavis 3939 - - (03939-17) Passed CLEAN {RelayedInternal}, MYNETS LOCAL [192.168.1.16]:43796 <cer@isengard.valinor> -> <cer@telcontar.valinor>, Queue-ID: A23EE3213B4, Message-ID: <alpine.LSU.2.21.1909031614120.15224@isengard.valinor>, mail_id: v-4c8WQA_QSF, Hits: -, size: 3365, queued_as: 7DD473213B7, 816 ms <2.6> 2019-09-03 16:24:44 Telcontar postfix 22507 - - A23EE3213B4: to=<cer@Telcontar.valinor>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.87, delays=0.01/0.04/0/0.82, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7DD473213B7) Wonderful! :-D -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Per Jessen wrote:
I think you can probably find a fairly low "working-set" that'll give you a decent response in 99% of cases.
Since yesterday, I have had one production cluster node running with clamd limited to 300Mb. Since midnight, approx 18000 out of 20000 emails were processed in less than 1sec. As the system has plenty of memory, I wonder if swapping is in fact very fast. A page might be marked as eligible for swap, but when that page isn't ever needed, swapping it back only means flipping that bit. I think I'll have to look at my test system instead. -- Per Jessen, Zürich (19.5°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/09/2019 14.37, Per Jessen wrote:
Per Jessen wrote:
I think you can probably find a fairly low "working-set" that'll give you a decent response in 99% of cases.
Since yesterday, I have had one production cluster node running with clamd limited to 300Mb. Since midnight, approx 18000 out of 20000 emails were processed in less than 1sec.
As the system has plenty of memory, I wonder if swapping is in fact very fast. A page might be marked as eligible for swap, but when that page isn't ever needed, swapping it back only means flipping that bit. I think I'll have to look at my test system instead.
My swap is in flash. Reading from swap is way faster than clamd reading the data from its own files. Surely they could adapt the code to create an image file internally, which could also be used for fast start. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
participants (2)
-
Carlos E. R. -
Per Jessen