Am 2. November 2021 13:48:54 MEZ schrieb Per Jessen :

* mailman also has an on-going issue with some posts, something about special characters. I think this most probably affects the archiving. issue #101791

Thankfully that's a frontend issue, looking at the hyperkitty database reveals everything is fine there

* more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap.

It would likely be a good idea to use something else than uwsgi, since that doesn't have the best performance. I may have a look into alternative solutions, but can't promise anything LCP [Sasi] https://lcp.world/

On 11/2/21 14:10, Neal Gompa wrote:

On Tue, Nov 2, 2021 at 9:07 AM Sasi Olin wrote:

...

Am 2. November 2021 13:48:54 MEZ schrieb Per Jessen :

...

* more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap.

It would likely be a good idea to use something else than uwsgi, since that doesn't have the best performance. I may have a look into alternative solutions, but can't promise anything >

Pagure uses gunicorn, maybe that's worth looking into for HyperKitty and Postorius?

Regarding the OOM kills I'm not entirely sure whether uwsgi is the culprit. My own test mailman installation required 1,6 GB of RAM while essentially doing nothing. FWIW I've also migrated all my web apps to run under control of gunicorn instead of uwsgi. The main reason was not performance but missing packages on some Linux distros (e.g. no uwsgi on CentOS8). Also you can remove mod_proxy_uwsgi from Apache config and use standard mod_proxy_http even if gunicorn listens on a Unix domain socket. However there's a subtle difference between gunicorn and uwsgi regarding when custom, non-worker *threads* are started. It required a change in my old-fashioned web app to deal with lazy thread initialization after initial gunicorn process fork. Likely mailman's worker processes are not affected by this though: https://mailman-bundler.readthedocs.io/en/stable/README.html#running-on-guni... It also took me a while to mentally accept that ProxyPass directive needs a pseudo IP address / port combo even if gunicorn only listens on a Unix domain socket - which is IMO pretty counterintuitive. Example: ProxyPass unix:/run/ae-dir/gunicorn/ae-dir-pwd/ae-dir-pwd.sock|http://127.0.0.1:8081/ If you're running into problems migrating to gunicorn feel free to contact me. Ciao, Michael.

On 11/2/21 15:28, Per Jessen wrote:

Michael Strc3b6der wrote:

...

On 11/2/21 14:10, Neal Gompa wrote:

...

On Tue, Nov 2, 2021 at 9:07 AM Sasi Olin wrote:

...

Am 2. November 2021 13:48:54 MEZ schrieb Per Jessen :

...

* more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap.

Regarding the OOM kills I'm not entirely sure whether uwsgi is the culprit. My own test mailman installation required 1,6 GB of RAM while essentially doing nothing.

Agree, I only mentioned uwsgi because that is what keeps getting killed.

Yes, I already understood you like that.

Right now, the uwsgi process is taking up 3Gb. We have some 6-7 nginx processes, each taking up about 800Mb, that is our 8Gb gone :-(

The nginx processes use so much RAM? Really? I'm asking because normally mailman's Python code is started by uwsgi (or gunicorn) and nginx (or Apache httpd) is just a lean reverse proxy in front of that. Ciao, Michael.

Sasi Olin wrote:

Am 2. November 2021 13:48:54 MEZ schrieb Per Jessen :

...

* mailman also has an on-going issue with some posts, something about special characters. I think this most probably affects the archiving. issue #101791

Thankfully that's a frontend issue, looking at the hyperkitty database reveals everything is fine there

Ah, okay. I never had time to look into it, at first glance it looked like something happened when mails were fed to the archive. I got diverted by the glibc issue.

...

* more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap.

It would likely be a good idea to use something else than uwsgi, since that doesn't have the best performance. I may have a look into alternative solutions, but can't promise anything

For now, I've asked Lars to increase the swap space - I really think 8Gb ought to be enough, but let's see. -- Per Jessen, Zürich (10.2°C) Member, openSUSE Heroes

Bernhard M. Wiedemann wrote:

On 02/11/2021 15.03, Per Jessen wrote:

...

For now, I've asked Lars to increase the swap space - I really think 8Gb ought to be enough, but let's see.

A larger swap can be a bad idea in many cases. Because if something starts to actively use that much memory, then it will keep writing to swap and reading from swap and that can kill performance. That is called "Thrashing"

Yeah, I know, I just thought 256M was really a bit too minimal. -- Per Jessen, Zürich (8.2°C) Member, openSUSE Heroes