Hello, the next heroes meeting will be on Tuesday (2021-11-02) at 19:00 UTC / 20:00 CET in https://meet.opensuse.org/heroes NOTE: Europe switched to CET today, therefore the UTC time changed to 19:00. We already have some topics and status updates listed on https://progress.opensuse.org/issues/99804 As usual, feel free to add yours ;-) Regards, Christian Boltz -- Vermutlich wird noch heute über Suizide namhafter Kabarettisten berichtet. Ihr Leben dürfte ab sofort überhaupt keinen Sinn mehr haben. Nur Zeitung lesen ist einfach witziger. [http://www.lawblog.de/index.php/archives/2014/07/04/die-armen-kabarettisten/]
Christian Boltz wrote:
Hello,
the next heroes meeting will be on Tuesday (2021-11-02) at 19:00 UTC / 20:00 CET in https://meet.opensuse.org/heroes
NOTE: Europe switched to CET today, therefore the UTC time changed to 19:00.
I have to excuse myself again, as usual. Brief status update - * mailman archive search is currently broken, I have been looking at it. An update of glibc broke xapian, but this is not necessarily directly to blame. issue #100802. * mailman also has an on-going issue with some posts, something about special characters. I think this most probably affects the archiving. issue #101791 * Olav and I continue to work on the saltification of mx12 - next step is a histate test-run (on mx-test.i.o.o). * I have also been working on getting widehat fully working. * there is a bunch of miscellaneous mirror tickets, new mirrors, faulty mirrors. * more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap. * I need to get LetsEncrypt certs automatically distributed to mx12, but apparently I don't have access to crtmgr. I'll open a ticket. -- Per Jessen, Zürich (13.4°C) Member, openSUSE Heroes
Am 2. November 2021 13:48:54 MEZ schrieb Per Jessen <per@opensuse.org>:
* mailman also has an on-going issue with some posts, something about special characters. I think this most probably affects the archiving. issue #101791
Thankfully that's a frontend issue, looking at the hyperkitty database reveals everything is fine there
* more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap.
It would likely be a good idea to use something else than uwsgi, since that doesn't have the best performance. I may have a look into alternative solutions, but can't promise anything LCP [Sasi] https://lcp.world/
On Tue, Nov 2, 2021 at 9:07 AM Sasi Olin <hellcp@opensuse.org> wrote:
Am 2. November 2021 13:48:54 MEZ schrieb Per Jessen <per@opensuse.org>:
* mailman also has an on-going issue with some posts, something about special characters. I think this most probably affects the archiving. issue #101791
Thankfully that's a frontend issue, looking at the hyperkitty database reveals everything is fine there
* more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap.
It would likely be a good idea to use something else than uwsgi, since that doesn't have the best performance. I may have a look into alternative solutions, but can't promise anything
Pagure uses gunicorn, maybe that's worth looking into for HyperKitty and Postorius? -- 真実はいつも一つ！/ Always, there's only one truth!
On 11/2/21 14:10, Neal Gompa wrote:
On Tue, Nov 2, 2021 at 9:07 AM Sasi Olin <hellcp@opensuse.org> wrote:
Am 2. November 2021 13:48:54 MEZ schrieb Per Jessen <per@opensuse.org>:
* more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap.
It would likely be a good idea to use something else than uwsgi, since that doesn't have the best performance. I may have a look into alternative solutions, but can't promise anything >
Pagure uses gunicorn, maybe that's worth looking into for HyperKitty and Postorius?
Regarding the OOM kills I'm not entirely sure whether uwsgi is the culprit. My own test mailman installation required 1,6 GB of RAM while essentially doing nothing. FWIW I've also migrated all my web apps to run under control of gunicorn instead of uwsgi. The main reason was not performance but missing packages on some Linux distros (e.g. no uwsgi on CentOS8). Also you can remove mod_proxy_uwsgi from Apache config and use standard mod_proxy_http even if gunicorn listens on a Unix domain socket. However there's a subtle difference between gunicorn and uwsgi regarding when custom, non-worker *threads* are started. It required a change in my old-fashioned web app to deal with lazy thread initialization after initial gunicorn process fork. Likely mailman's worker processes are not affected by this though: https://mailman-bundler.readthedocs.io/en/stable/README.html#running-on-guni... It also took me a while to mentally accept that ProxyPass directive needs a pseudo IP address / port combo even if gunicorn only listens on a Unix domain socket - which is IMO pretty counterintuitive. Example: ProxyPass unix:/run/ae-dir/gunicorn/ae-dir-pwd/ae-dir-pwd.sock|http://127.0.0.1:8081/ If you're running into problems migrating to gunicorn feel free to contact me. Ciao, Michael.
Michael Strc3b6der wrote:
On 11/2/21 14:10, Neal Gompa wrote:
On Tue, Nov 2, 2021 at 9:07 AM Sasi Olin <hellcp@opensuse.org> wrote:
Am 2. November 2021 13:48:54 MEZ schrieb Per Jessen <per@opensuse.org>:
* more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap.
Regarding the OOM kills I'm not entirely sure whether uwsgi is the culprit. My own test mailman installation required 1,6 GB of RAM while essentially doing nothing.
Agree, I only mentioned uwsgi because that is what keeps getting killed. Right now, the uwsgi process is taking up 3Gb. We have some 6-7 nginx processes, each taking up about 800Mb, that is our 8Gb gone :-( -- Per Jessen, Zürich (13.1°C) Member, openSUSE Heroes
On 11/2/21 15:28, Per Jessen wrote:
Michael Strc3b6der wrote:
On 11/2/21 14:10, Neal Gompa wrote:
On Tue, Nov 2, 2021 at 9:07 AM Sasi Olin <hellcp@opensuse.org> wrote:
Am 2. November 2021 13:48:54 MEZ schrieb Per Jessen <per@opensuse.org>:
* more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap.
Regarding the OOM kills I'm not entirely sure whether uwsgi is the culprit. My own test mailman installation required 1,6 GB of RAM while essentially doing nothing.
Agree, I only mentioned uwsgi because that is what keeps getting killed.
Yes, I already understood you like that.
Right now, the uwsgi process is taking up 3Gb. We have some 6-7 nginx processes, each taking up about 800Mb, that is our 8Gb gone :-(
The nginx processes use so much RAM? Really? I'm asking because normally mailman's Python code is started by uwsgi (or gunicorn) and nginx (or Apache httpd) is just a lean reverse proxy in front of that. Ciao, Michael.
Michael Strc3b6der wrote:
On 11/2/21 15:28, Per Jessen wrote:
Michael Strc3b6der wrote:
On 11/2/21 14:10, Neal Gompa wrote:
On Tue, Nov 2, 2021 at 9:07 AM Sasi Olin <hellcp@opensuse.org> wrote:
Am 2. November 2021 13:48:54 MEZ schrieb Per Jessen <per@opensuse.org>:
* more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap.
Regarding the OOM kills I'm not entirely sure whether uwsgi is the culprit. My own test mailman installation required 1,6 GB of RAM while essentially doing nothing.
Agree, I only mentioned uwsgi because that is what keeps getting killed.
Yes, I already understood you like that.
Right now, the uwsgi process is taking up 3Gb. We have some 6-7 nginx processes, each taking up about 800Mb, that is our 8Gb gone :-(
The nginx processes use so much RAM? Really?
It is probably because of our huge redirmaps (3mill entries) from the old archives to the new archives. I've opened a ticket, I think there is room for improvement, #101842 -- Per Jessen, Zürich (11.2°C) Member, openSUSE Heroes
Sasi Olin wrote:
Am 2. November 2021 13:48:54 MEZ schrieb Per Jessen <per@opensuse.org>:
* mailman also has an on-going issue with some posts, something about special characters. I think this most probably affects the archiving. issue #101791
Thankfully that's a frontend issue, looking at the hyperkitty database reveals everything is fine there
Ah, okay. I never had time to look into it, at first glance it looked like something happened when mails were fed to the archive. I got diverted by the glibc issue.
* more mailman stuff - 'uwsgi' keeps getting killed by the oom killer, 3-4-5 times a day. The machine has 8Gb, but very little swap.
It would likely be a good idea to use something else than uwsgi, since that doesn't have the best performance. I may have a look into alternative solutions, but can't promise anything
For now, I've asked Lars to increase the swap space - I really think 8Gb ought to be enough, but let's see. -- Per Jessen, Zürich (10.2°C) Member, openSUSE Heroes
On 02/11/2021 15.03, Per Jessen wrote:
For now, I've asked Lars to increase the swap space - I really think 8Gb ought to be enough, but let's see.
A larger swap can be a bad idea in many cases. Because if something starts to actively use that much memory, then it will keep writing to swap and reading from swap and that can kill performance. That is called "Thrashing" Atm I see 7 nginx processes at 1GB RAM usage, but they are forked from a common parent, so much of that RAM is shared between them. It can happen though, that if data changes, it gets unshared over time.
Bernhard M. Wiedemann wrote:
On 02/11/2021 15.03, Per Jessen wrote:
For now, I've asked Lars to increase the swap space - I really think 8Gb ought to be enough, but let's see.
A larger swap can be a bad idea in many cases. Because if something starts to actively use that much memory, then it will keep writing to swap and reading from swap and that can kill performance. That is called "Thrashing"
Yeah, I know, I just thought 256M was really a bit too minimal. -- Per Jessen, Zürich (8.2°C) Member, openSUSE Heroes
participants (6)
-
Bernhard M. Wiedemann -
Christian Boltz -
Michael Ströder -
Neal Gompa -
Per Jessen -
Sasi Olin