[Bug 1222450] New: Error: crun: opening file `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found
https://bugzilla.suse.com/show_bug.cgi?id=1222450 Bug ID: 1222450 Summary: Error: crun: opening file `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Containers Assignee: containers-bugowner@suse.de Reporter: miika.alikirri@suse.com QA Contact: qa-bugs@suse.de Target Milestone: --- Found By: --- Blocker: --- Created attachment 874124 --> https://bugzilla.suse.com/attachment.cgi?id=874124&action=edit output of podman info On a fresh installation of tumbleweed (version 20240403), podman fails to run as a non-root user when `--memory` option is set. running: ``` podman run --rm --memory 4M tumbleweed echo it works ``` prints out: ``` Error: crun: opening file `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found ``` And running it as sudo runs it correctly and prints: ``` it works ``` -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 https://bugzilla.suse.com/show_bug.cgi?id=1222450#c1 Aleksa Sarai <asarai@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(miika.alikirri@su | |se.com) CC| |asarai@suse.com, | |miika.alikirri@suse.com --- Comment #1 from Aleksa Sarai <asarai@suse.com> --- It seems that your machine doesn't have the memory controller enabled on cgroupv2 (in fact it seems to only have pids enabled). What is the contents of /sys/fs/cgroup/cgroup.controllers? cgroupControllers: - pids cgroupManager: systemd cgroupVersion: v2 Because systemd only allows unprivileged users to manage cgroupv2 cgroups, crun can't do whatever cgroupv1 fallback it is presumably doing when you run podman as root. (FWIW, I can't reproduce this at all on my Tumbleweed machine. It's not a fresh install though.) -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 Aleksa Sarai <asarai@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|containers-bugowner@suse.de |asarai@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 https://bugzilla.suse.com/show_bug.cgi?id=1222450#c2 --- Comment #2 from Aleksa Sarai <asarai@suse.com> --- (In reply to Aleksa Sarai from comment #1)
What is the contents of /sys/fs/cgroup/cgroup.controllers?
And also /sys/fs/cgroup/cgroup.subtree_control -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 https://bugzilla.suse.com/show_bug.cgi?id=1222450#c3 Miika Alikirri <miika.alikirri@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(miika.alikirri@su | |se.com) | --- Comment #3 from Miika Alikirri <miika.alikirri@suse.com> --- (In reply to Aleksa Sarai from comment #2)
(In reply to Aleksa Sarai from comment #1)
What is the contents of /sys/fs/cgroup/cgroup.controllers?
And also /sys/fs/cgroup/cgroup.subtree_control
I downloaded a latest snapshot and did nothing but install podman on it and got the same result. Here are the file contents: $ cat /sys/fs/cgroup/cgroup.subtree_control pids $ cat /sys/fs/cgroup/cgroup.controllers cpuset cpu io memory hugetlb pids rdma misc -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 https://bugzilla.suse.com/show_bug.cgi?id=1222450#c4 --- Comment #4 from Miika Alikirri <miika.alikirri@suse.com> --- Any progress on this? Is there anything I can do to help? -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 https://bugzilla.suse.com/show_bug.cgi?id=1222450#c5 Aleksa Sarai <asarai@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|Containers |Basesystem Assignee|asarai@suse.com |screening-team-bugs@suse.de --- Comment #5 from Aleksa Sarai <asarai@suse.com> --- So, the issue is that systemd is not enabling all of the controllers at the root (subtree_control is missing a bunch of controllers on your system), resulting in the container not being able to start because the controllers are not available. This is a little surprising because crun/runc creates a systemd transient unit that has all of the restrictions listed. I can reproduce this somewhat on my machine -- depending on the running services, the set of subtree_control-enabled controllers varies. If I start Docker and run a container, all of the controllers are enabled and using podman works. But if you stop the Docker service, the cpu controller is disabled and so --cpu-shares no longer works. This seems like a systemd issue to me. Why is a TransientUnit with all of the relevant restrictions applied not sufficient to get systemd to enable the needed controllers? -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 https://bugzilla.suse.com/show_bug.cgi?id=1222450#c6 Chenzi Cao <chcao@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|screening-team-bugs@suse.de |systemd-maintainers@suse.de --- Comment #6 from Chenzi Cao <chcao@suse.com> --- I assign it to systemd maintainers to take a look at this, please feel free to reassign whenever necessary, thanks. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 https://bugzilla.suse.com/show_bug.cgi?id=1222450#c7 Franck Bui <fbui@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fbui@suse.com --- Comment #7 from Franck Bui <fbui@suse.com> --- (In reply to Aleksa Sarai from comment #5)
So, the issue is that systemd is not enabling all of the controllers at the root (subtree_control is missing a bunch of controllers on your system),
AFAIK the controllers are usually enabled on demand, ie if no unit requires them then they remain disabled. It's actually important because usually (some) cgroup controllers incur some performance penalties to the whole system even if only one single unit needs them. That's for this reasons that it was requested to leave most of the controllers disabled by default: --- %< --- # rpm -q --changelog systemd-default-settings | head -n7 * Wed Apr 10 2024 Franck Bui <fbui@suse.com> - Import 0.10 5088997 SLE: Disable pids controller limit under user instances (jsc#SLE-10123) * Thu Mar 28 2024 Franck Bui <fbui@suse.com> - Import 0.9 bb859bf user@.service: Disable controllers by default (jsc#PED-2276) --- >% ---- -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 https://bugzilla.suse.com/show_bug.cgi?id=1222450#c8 Franck Bui <fbui@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(asarai@suse.com) --- Comment #8 from Franck Bui <fbui@suse.com> --- (In reply to Aleksa Sarai from comment #5)
This seems like a systemd issue to me. Why is a TransientUnit with all of the relevant restrictions applied not sufficient to get systemd to enable the needed controllers?
Which transient unit ? Are you meaning that `podman run` creates a transient unit with some memory constraints set ? I can't find any trace of it when running `podman run ...` -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 https://bugzilla.suse.com/show_bug.cgi?id=1222450#c9 Franck Bui <fbui@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|systemd-maintainers@suse.de |containers-bugowner@suse.de --- Comment #9 from Franck Bui <fbui@suse.com> --- Let's reassign this back to the container team for the moment. I don't think systemd behaves incorrectly: if a service needs to have access to a cgroup controller it needs to explicitly enable it and not assume it's always enabled. Feel free to reassign it back to us in case I misunderstood the problem. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 https://bugzilla.suse.com/show_bug.cgi?id=1222450#c10 Aleksa Sarai <asarai@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(asarai@suse.com) |needinfo?(fbui@suse.com) --- Comment #10 from Aleksa Sarai <asarai@suse.com> --- (In reply to Franck Bui from comment #8)
(In reply to Aleksa Sarai from comment #5)
This seems like a systemd issue to me. Why is a TransientUnit with all of the relevant restrictions applied not sufficient to get systemd to enable the needed controllers?
Which transient unit ?
Are you meaning that `podman run` creates a transient unit with some memory constraints set ?
I can't find any trace of it when running `podman run ...`
runc creates transient units based on the user configuration[1] and it seems that crun does too[2]. I just tried to start some containers using runc and it seems that the TransientUnit we configure isn't sufficient? `podman run` (without sudo) cannot create cgroups by itself so it must be contacting systemd to create the cgroups with a TransientUnit. What command can I run to get information about existing TransientUnits? (As an aside I don't know why we ship crun at all -- runc is the runtime that we support in general in SLES, and I don't get why we would ship a second runtime just for openSUSE and just for podman.) [1]: https://github.com/opencontainers/runc/blob/main/libcontainer/cgroups/system... [2]: https://github.com/containers/crun/blob/e6a1ef18c5f313b0b6c4e4ee85688f80ff35... -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 Aleksa Sarai <asarai@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|containers-bugowner@suse.de |asarai@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 Aleksa Sarai <asarai@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |containers-bugowner@suse.de -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1222450 https://bugzilla.suse.com/show_bug.cgi?id=1222450#c11 Franck Bui <fbui@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(fbui@suse.com) | --- Comment #11 from Franck Bui <fbui@suse.com> --- (In reply to Aleksa Sarai from comment #10)
What command can I run to get information about existing TransientUnits?
(sorry for the delay). Perhaps the easiest way is to run `systemd-cgls` and to look for the transient unit in the scope/slice that it's been supposed to run in. -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com