Creating a cgroup, I need a 2nd pair of eyes
On leap 15.2, I am creating a cgroup: cd /sys/fs/cgroup/memory mkdir hpindexer echo 2147483648 >hpindexer/memory.limit_in_bytes echo 17493 >hpindexer/cgroup.procs At first it seems to work fine, but after a certain time, "hpindexer" just disappears ... What am I missing ? -- Per Jessen, Zürich (-0.6°C) Member, openSUSE Heroes.
On 07/01/2021 14.12, Per Jessen wrote:
On leap 15.2, I am creating a cgroup:
cd /sys/fs/cgroup/memory mkdir hpindexer echo 2147483648 >hpindexer/memory.limit_in_bytes echo 17493 >hpindexer/cgroup.procs
At first it seems to work fine, but after a certain time, "hpindexer" just disappears ...
What am I missing ?
Is it run by systemd? -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
Carlos E.R. wrote:
On 07/01/2021 14.12, Per Jessen wrote:
On leap 15.2, I am creating a cgroup:
cd /sys/fs/cgroup/memory mkdir hpindexer echo 2147483648 >hpindexer/memory.limit_in_bytes echo 17493 >hpindexer/cgroup.procs
At first it seems to work fine, but after a certain time, "hpindexer" just disappears ...
What am I missing ?
Is it run by systemd?
No, it's just a manually started process, with a manually created cgroup. Otherwise I would I have left it to systemd to create the cgroup. -- Per Jessen, Zürich (0.4°C)
On 07/01/2021 14.21, Per Jessen wrote:
Carlos E.R. wrote:
On 07/01/2021 14.12, Per Jessen wrote:
On leap 15.2, I am creating a cgroup:
cd /sys/fs/cgroup/memory mkdir hpindexer echo 2147483648 >hpindexer/memory.limit_in_bytes echo 17493 >hpindexer/cgroup.procs
At first it seems to work fine, but after a certain time, "hpindexer" just disappears ...
What am I missing ?
Is it run by systemd?
No, it's just a manually started process, with a manually created cgroup. Otherwise I would I have left it to systemd to create the cgroup.
Ah, then I don't have a clue, sorry. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
07.01.2021 16:21, Per Jessen пишет:
Carlos E.R. wrote:
On 07/01/2021 14.12, Per Jessen wrote:
On leap 15.2, I am creating a cgroup:
cd /sys/fs/cgroup/memory mkdir hpindexer echo 2147483648 >hpindexer/memory.limit_in_bytes echo 17493 >hpindexer/cgroup.procs
At first it seems to work fine, but after a certain time, "hpindexer" just disappears ...
How long is "certain time"?
What am I missing ?
Is it run by systemd?
No, it's just a manually started process,
How exactly are you starting it?
with a manually created cgroup. Otherwise I would I have left it to systemd to create the cgroup.
Andrei Borzenkov wrote:
07.01.2021 16:21, Per Jessen пишет:
Carlos E.R. wrote:
On 07/01/2021 14.12, Per Jessen wrote:
On leap 15.2, I am creating a cgroup:
cd /sys/fs/cgroup/memory mkdir hpindexer echo 2147483648 >hpindexer/memory.limit_in_bytes echo 17493 >hpindexer/cgroup.procs
At first it seems to work fine, but after a certain time, "hpindexer" just disappears ...
How long is "certain time"?
It seems to vary - sometimes 10mins, but right now it's been almost an hour. I have a loop checking on hpindexer.memory.usage_in_bytes - I suspect that has changed the behaviour - heisenbug.
What am I missing ?
Is it run by systemd?
No, it's just a manually started process,
How exactly are you starting it?
Just with nohup and '&'. nohup python3 ..... & -- Per Jessen, Zürich (0.2°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland.
Per Jessen wrote:
Andrei Borzenkov wrote:
07.01.2021 16:21, Per Jessen пишет:
Carlos E.R. wrote:
On 07/01/2021 14.12, Per Jessen wrote:
On leap 15.2, I am creating a cgroup:
cd /sys/fs/cgroup/memory mkdir hpindexer echo 2147483648 >hpindexer/memory.limit_in_bytes echo 17493 >hpindexer/cgroup.procs
At first it seems to work fine, but after a certain time, "hpindexer" just disappears ...
How long is "certain time"?
It seems to vary - sometimes 10mins, but right now it's been almost an hour.
The process kept running overnight, but the cgroup disappeared again. I expect the process to maybe take a few more days. I recreated the cgroup this morning, for now it has remained for a little over an hour. I have no idea if this has any significance - the timestamps of the many settings in /sys/fs/cgroup/memory "behave" differently on Leap 15.1 and 15.2 : Leap 15.1 - on a system that has been up for 50 days, the timestamps first read "Nov 24 16:59" before I created a cgroup. After creation, the timestamps read "Jan 8 10:06" and they remain. Leap 15.2 - the timestamps are always current, they appear to be continually updated: # ls --full-time /sys/fs/cgroup/memory total 0 -rw-r--r-- 1 root root 0 2021-01-08 09:19:38.465817860 +0000 cgroup.clone_children --w--w--w- 1 root root 0 2021-01-08 09:19:38.465817860 +0000 cgroup.event_control -rw-r--r-- 1 root root 0 2021-01-08 09:19:38.465817860 +0000 cgroup.procs -r--r--r-- 1 root root 0 2021-01-08 09:19:38.465817860 +0000 cgroup.sane_behavior drwxr-xr-x 2 root root 0 2021-01-08 09:19:38.465817860 +0000 hpindexer [snip] # ls --full-time /sys/fs/cgroup/memory total 0 -rw-r--r-- 1 root root 0 2021-01-08 09:19:42.585674921 +0000 cgroup.clone_children --w--w--w- 1 root root 0 2021-01-08 09:19:42.585674921 +0000 cgroup.event_control -rw-r--r-- 1 root root 0 2021-01-08 09:19:42.581675059 +0000 cgroup.procs -r--r--r-- 1 root root 0 2021-01-08 09:19:42.585674921 +0000 cgroup.sane_behavior drwxr-xr-x 2 root root 0 2021-01-08 09:19:42.585674921 +0000 hpindexer [snip] # ls --full-time /sys/fs/cgroup/memory total 0 -rw-r--r-- 1 root root 0 2021-01-08 09:19:48.589466618 +0000 cgroup.clone_children --w--w--w- 1 root root 0 2021-01-08 09:19:48.589466618 +0000 cgroup.event_control -rw-r--r-- 1 root root 0 2021-01-08 09:19:48.589466618 +0000 cgroup.procs -r--r--r-- 1 root root 0 2021-01-08 09:19:48.589466618 +0000 cgroup.sane_behavior drwxr-xr-x 2 root root 0 2021-01-08 09:19:48.589466618 +0000 hpindexer [snip] -- Per Jessen, Zürich (-0.1°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.
08.01.2021 12:24, Per Jessen пишет:
Per Jessen wrote:
Andrei Borzenkov wrote:
07.01.2021 16:21, Per Jessen пишет:
Carlos E.R. wrote:
On 07/01/2021 14.12, Per Jessen wrote:
On leap 15.2, I am creating a cgroup:
cd /sys/fs/cgroup/memory mkdir hpindexer echo 2147483648 >hpindexer/memory.limit_in_bytes echo 17493 >hpindexer/cgroup.procs
At first it seems to work fine, but after a certain time, "hpindexer" just disappears ...
How long is "certain time"?
It seems to vary - sometimes 10mins, but right now it's been almost an hour.
The process kept running overnight, but the cgroup disappeared again. I expect the process to maybe take a few more days.
Did you logout over night? I left VM running for several hours with active login session, process was started in this session, cgroup was preserved.
Andrei Borzenkov wrote:
08.01.2021 12:24, Per Jessen пишет:
Per Jessen wrote:
Andrei Borzenkov wrote:
07.01.2021 16:21, Per Jessen пишет:
Carlos E.R. wrote:
On 07/01/2021 14.12, Per Jessen wrote: > On leap 15.2, I am creating a cgroup: > > cd /sys/fs/cgroup/memory > mkdir hpindexer > echo 2147483648 >hpindexer/memory.limit_in_bytes > echo 17493 >hpindexer/cgroup.procs > > At first it seems to work fine, but after a certain time, > "hpindexer" just disappears ... >
How long is "certain time"?
It seems to vary - sometimes 10mins, but right now it's been almost an hour.
The process kept running overnight, but the cgroup disappeared again. I expect the process to maybe take a few more days.
Did you logout over night? I left VM running for several hours with active login session, process was started in this session, cgroup was preserved.
Yes, I was logged in from a laptop which I closed overnight, so the network was cut off. -- Per Jessen, Zürich (0.9°C)
08.01.2021 12:49, Per Jessen пишет:
Andrei Borzenkov wrote:
08.01.2021 12:24, Per Jessen пишет:
Per Jessen wrote:
Andrei Borzenkov wrote:
07.01.2021 16:21, Per Jessen пишет:
Carlos E.R. wrote:
> On 07/01/2021 14.12, Per Jessen wrote: >> On leap 15.2, I am creating a cgroup: >> >> cd /sys/fs/cgroup/memory >> mkdir hpindexer >> echo 2147483648 >hpindexer/memory.limit_in_bytes >> echo 17493 >hpindexer/cgroup.procs >> >> At first it seems to work fine, but after a certain time, >> "hpindexer" just disappears ... >>
How long is "certain time"?
It seems to vary - sometimes 10mins, but right now it's been almost an hour.
The process kept running overnight, but the cgroup disappeared again. I expect the process to maybe take a few more days.
Did you logout over night? I left VM running for several hours with active login session, process was started in this session, cgroup was preserved.
Yes, I was logged in from a laptop which I closed overnight, so the network was cut off.
In this case it could be side effect of systemd cleaning up user session after logout.
Andrei Borzenkov wrote:
08.01.2021 12:49, Per Jessen пишет:
Andrei Borzenkov wrote:
08.01.2021 12:24, Per Jessen пишет:
Per Jessen wrote:
Andrei Borzenkov wrote:
07.01.2021 16:21, Per Jessen пишет: > Carlos E.R. wrote: > >> On 07/01/2021 14.12, Per Jessen wrote: >>> On leap 15.2, I am creating a cgroup: >>> >>> cd /sys/fs/cgroup/memory >>> mkdir hpindexer >>> echo 2147483648 >hpindexer/memory.limit_in_bytes >>> echo 17493 >hpindexer/cgroup.procs >>> >>> At first it seems to work fine, but after a certain time, >>> "hpindexer" just disappears ... >>>
How long is "certain time"?
It seems to vary - sometimes 10mins, but right now it's been almost an hour.
The process kept running overnight, but the cgroup disappeared again. I expect the process to maybe take a few more days.
Did you logout over night? I left VM running for several hours with active login session, process was started in this session, cgroup was preserved.
Yes, I was logged in from a laptop which I closed overnight, so the network was cut off.
In this case it could be side effect of systemd cleaning up user session after logout.
Hmm, that's not good. I have just done this experiment - log in 'per' over ssh su root cd /sys/fs/cgroup/memory create cgroup etc. leave root logout login again - cgroup is gone. (repeated twice, same thing). -- Per Jessen, Zürich (1.1°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland.
On Fri, 08 Jan 2021 11:23:01 +0100 Per Jessen <per@computer.org> wrote:
Andrei Borzenkov wrote:
08.01.2021 12:49, Per Jessen пишет:
Andrei Borzenkov wrote:
08.01.2021 12:24, Per Jessen пишет:
Per Jessen wrote:
Andrei Borzenkov wrote:
> 07.01.2021 16:21, Per Jessen пишет: >> Carlos E.R. wrote: >> >>> On 07/01/2021 14.12, Per Jessen wrote: >>>> On leap 15.2, I am creating a cgroup: >>>> >>>> cd /sys/fs/cgroup/memory >>>> mkdir hpindexer >>>> echo 2147483648 >hpindexer/memory.limit_in_bytes >>>> echo 17493 >hpindexer/cgroup.procs >>>> >>>> At first it seems to work fine, but after a certain time, >>>> "hpindexer" just disappears ... >>>> > > How long is "certain time"?
It seems to vary - sometimes 10mins, but right now it's been almost an hour.
The process kept running overnight, but the cgroup disappeared again. I expect the process to maybe take a few more days.
Did you logout over night? I left VM running for several hours with active login session, process was started in this session, cgroup was preserved.
Yes, I was logged in from a laptop which I closed overnight, so the network was cut off.
In this case it could be side effect of systemd cleaning up user session after logout.
Hmm, that's not good. I have just done this experiment -
log in 'per' over ssh su root cd /sys/fs/cgroup/memory create cgroup etc. leave root logout
login again - cgroup is gone.
(repeated twice, same thing).
Don't know anything about the subject but ... Maybe systemd-run --scope ? Based on what I skimmed in https://unix.stackexchange.com/questions/35215/creating-one-time-cgroup https://www.freedesktop.org/software/systemd/man/systemd-run.html https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/
Dave Howorth wrote:
Don't know anything about the subject but ...
Maybe systemd-run --scope ?
Based on what I skimmed in https://unix.stackexchange.com/questions/35215/creating-one-time-cgroup https://www.freedesktop.org/software/systemd/man/systemd-run.html
Hmm, I didn't know that one, but do I really need systemd in the "mix" ? -- Per Jessen, Zürich (1.6°C)
On Fri, 08 Jan 2021 12:38:19 +0100 Per Jessen <per@computer.org> wrote:
Dave Howorth wrote:
Don't know anything about the subject but ...
Maybe systemd-run --scope ?
Based on what I skimmed in https://unix.stackexchange.com/questions/35215/creating-one-time-cgroup https://www.freedesktop.org/software/systemd/man/systemd-run.html
Hmm, I didn't know that one, but do I really need systemd in the "mix" ?
Dunno, but maybe it gives you a way to start the job with a lifetime that's not tied to your ssh login?
On 1/8/21 10:09 AM, Dave Howorth wrote:
On Fri, 08 Jan 2021 12:38:19 +0100 Per Jessen <per@computer.org> wrote:
Dave Howorth wrote:
Don't know anything about the subject but ...
Maybe systemd-run --scope ?
Based on what I skimmed in https://unix.stackexchange.com/questions/35215/creating-one-time-cgroup https://www.freedesktop.org/software/systemd/man/systemd-run.html
Hmm, I didn't know that one, but do I really need systemd in the "mix" ?
Dunno, but maybe it gives you a way to start the job with a lifetime that's not tied to your ssh login?
Might screen work? Regards Mark
Dave Howorth wrote:
On Fri, 08 Jan 2021 12:38:19 +0100 Per Jessen <per@computer.org> wrote:
Dave Howorth wrote:
Don't know anything about the subject but ...
Maybe systemd-run --scope ?
Based on what I skimmed in
https://unix.stackexchange.com/questions/35215/creating-one-time-cgroup
https://www.freedesktop.org/software/systemd/man/systemd-run.html
Hmm, I didn't know that one, but do I really need systemd in the "mix" ?
Dunno, but maybe it gives you a way to start the job with a lifetime that's not tied to your ssh login?
That is already achieved with the 'nohup'. The job is not the problem, it keeps running, the only problem is the disappearing cgroup. Maybe I will have to try it with systemd-run - the job was oom killed about half an hour ago ... after more than 24 hours :-( The cgroup had probably been removed when we had a slight hiccup on the openSUSE vpn. -- Per Jessen, Zürich (-2.1°C) Member, openSUSE Heroes
Per Jessen wrote:
Maybe I will have to try it with systemd-run -
Have just restarted with this: systemd-run --unit=hpindexer \ --property=WorkingDirectory=/var/blahblah \ --property=MemoryHigh=2G \ --uid-mailman --gid=mailman \ python3 .... First impressions - looks good, I also see a cgroup created: /sys/fs/cgroup/memory/system.slice/hpindexer.service What I don't see is any 2Gb memory limit .... -- Per Jessen, Zürich (-2.1°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland.
08.01.2021 18:59, Per Jessen пишет:
Per Jessen wrote:
Maybe I will have to try it with systemd-run -
Have just restarted with this:
systemd-run --unit=hpindexer \ --property=WorkingDirectory=/var/blahblah \ --property=MemoryHigh=2G \ --uid-mailman --gid=mailman \ python3 ....
First impressions - looks good, I also see a cgroup created:
/sys/fs/cgroup/memory/system.slice/hpindexer.service
What I don't see is any 2Gb memory limit ....
MemoryHigh is supported only on unified cgroup hierarchy (cgroup v2). Use MemoryLimit which maps to memory.limit_in_bytes.
Per Jessen wrote:
Per Jessen wrote:
Maybe I will have to try it with systemd-run -
Have just restarted with this:
systemd-run --unit=hpindexer \ --property=WorkingDirectory=/var/blahblah \ --property=MemoryHigh=2G \ --uid-mailman --gid=mailman \ python3 ....
First impressions - looks good, I also see a cgroup created:
/sys/fs/cgroup/memory/system.slice/hpindexer.service
What I don't see is any 2Gb memory limit ....
Seems I should have used 'MemoryMax' instead. That solves the immediate issue, will be interesting to see if the process is now able to complete, maybe in a few days. Still, not being able to keep a manually created cgroup does not sound right. Next time, I will have forgotten all about systemd-run. -- Per Jessen, Zürich (-3.2°C)
On 08/01/2021 11.23, Per Jessen wrote:
Andrei Borzenkov wrote:
08.01.2021 12:49, Per Jessen пишет:
Andrei Borzenkov wrote:
08.01.2021 12:24, Per Jessen пишет:
...
Did you logout over night? I left VM running for several hours with active login session, process was started in this session, cgroup was preserved.
Yes, I was logged in from a laptop which I closed overnight, so the network was cut off.
In this case it could be side effect of systemd cleaning up user session after logout.
Hmm, that's not good. I have just done this experiment -
log in 'per' over ssh su root cd /sys/fs/cgroup/memory create cgroup etc. leave root logout
login again - cgroup is gone.
(repeated twice, same thing).
Wow. Would using tmux help? I don't see how, though, tmux would have to be started out of the ssh session. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
08.01.2021 13:23, Per Jessen пишет:
Andrei Borzenkov wrote:
08.01.2021 12:49, Per Jessen пишет:
Andrei Borzenkov wrote:
08.01.2021 12:24, Per Jessen пишет:
Per Jessen wrote:
Andrei Borzenkov wrote:
> 07.01.2021 16:21, Per Jessen пишет: >> Carlos E.R. wrote: >> >>> On 07/01/2021 14.12, Per Jessen wrote: >>>> On leap 15.2, I am creating a cgroup: >>>> >>>> cd /sys/fs/cgroup/memory >>>> mkdir hpindexer >>>> echo 2147483648 >hpindexer/memory.limit_in_bytes >>>> echo 17493 >hpindexer/cgroup.procs >>>> >>>> At first it seems to work fine, but after a certain time, >>>> "hpindexer" just disappears ... >>>> > > How long is "certain time"?
It seems to vary - sometimes 10mins, but right now it's been almost an hour.
The process kept running overnight, but the cgroup disappeared again. I expect the process to maybe take a few more days.
Did you logout over night? I left VM running for several hours with active login session, process was started in this session, cgroup was preserved.
Yes, I was logged in from a laptop which I closed overnight, so the network was cut off.
In this case it could be side effect of systemd cleaning up user session after logout.
Hmm, that's not good. I have just done this experiment -
log in 'per' over ssh su root cd /sys/fs/cgroup/memory create cgroup etc. leave root logout
login again - cgroup is gone.
(repeated twice, same thing).
Is only cgroup gone or process(es) it contained also?
Andrei Borzenkov wrote:
08.01.2021 13:23, Per Jessen пишет:
Hmm, that's not good. I have just done this experiment -
log in 'per' over ssh su root cd /sys/fs/cgroup/memory create cgroup etc. leave root logout
login again - cgroup is gone.
(repeated twice, same thing).
Is only cgroup gone or process(es) it contained also?
The process (just one) keeps running, no probs. It is only the cgroup that disappears. -- Per Jessen, Zürich (-2.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.
Per Jessen wrote:
On leap 15.2, I am creating a cgroup:
cd /sys/fs/cgroup/memory mkdir hpindexer echo 2147483648 >hpindexer/memory.limit_in_bytes echo 17493 >hpindexer/cgroup.procs
At first it seems to work fine, but after a certain time, "hpindexer" just disappears ...
What am I missing ?
Well. Don't you just love it when you give in and post a question, and next thing, the problem disappears. Heisenbug ... Of course, now that I have written and posted this, the problem will immediately re-appear. I need a beer. -- Per Jessen, Zürich (-0.2°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland.
On 1/7/21 6:45 PM, Per Jessen wrote:
Well. Don't you just love it when you give in and post a question, and next thing, the problem disappears. Heisenbug ...
Of course, now that I have written and posted this, the problem will immediately re-appear. I need a beer.
By concept, this will be a "Heisenbeer" - it will disappear while inspecting it as well. ;-)
participants (7)
-
Andrei Borzenkov -
Bernhard Voelker -
Carlos E. R. -
Carlos E.R. -
Dave Howorth -
Mark Hounschell -
Per Jessen