[Bug 1214845] New: libvirtError: Unable to find 'memory' cgroups controller mount
https://bugzilla.suse.com/show_bug.cgi?id=1214845 Bug ID: 1214845 Summary: libvirtError: Unable to find 'memory' cgroups controller mount Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: openSUSE Tumbleweed Status: NEW Severity: Normal Priority: P5 - None Component: Virtualization:Other Assignee: virt-bugs@suse.de Reporter: chrisvte@gmail.com QA Contact: qa-bugs@suse.de Target Milestone: --- Found By: --- Blocker: --- After I updated to kernel 6.4.11-1-default, every time I booted a LXC I was getting this error: “libvirtError: internal error: Unable to find ‘memory’ cgroups controller mount” I replicated the installation in a minipc (Intel N5105) and it worked perfectly, so I did a new installation in the first computer (AMD Ryzen 5600G) and, again, I got the same error of the beginning. I tried in other two computers (AMD Ryzen 5900X and Intel i5-1135G7) and I got the same error. If I added “systemd.unified_cgroup_hierarchy=0” to the boot line, the error went away, but it was really unstable (containers stopped working after a while). I usually install the tools of virtualization through Yast (KVM Server and KVM tools). After that I install a few more packages: libvirt-client, libvirt-daemon, libvirt-daemon-lxc, libvirt-daemon-driver-lxc, system-user-libvirt-dbus. Finally, I add the user to groups libvirt, qemu and kvm. Reboot and run. The details of the error using Virt-Manager: Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 72, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 108, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn ret = fn(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/virt-manager/virtManager/object/domain.py", line 1425, in startup self._backend.create() File "/usr/lib64/python3.11/site-packages/libvirt.py", line 1373, in create raise libvirtError('virDomainCreate() failed') libvirt.libvirtError: Unable to find 'memory' cgroups controller mount And using command line: # virsh -d 0 -c lxc:// start 101-PiHole start: domain(optdata): 101-PiHole start: found option <domain>: 101-PiHole start: <domain> trying as domain NAME error: Failed to start domain '101-PiHole' error: internal error: Unable to find 'memory' cgroups controller mount About the system: # mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) # uname -a Linux 6.4.11-1-default #1 SMP PREEMPT_DYNAMIC Thu Aug 17 04:57:43 UTC 2023 (2a5b3f6) x86_64 x86_64 x86_64 GNU/Linux -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214845 https://bugzilla.suse.com/show_bug.cgi?id=1214845#c1 --- Comment #1 from Chris <chrisvte@gmail.com> --- To mimic the minipc working system I installed this packages (with their dependencies): - patterns-microos-cockpit - cockpit - cockpit-bridge - cockpit-kdump - cockpit-machines - cockpit-networkmanager - cockpit-packagekit - cockpit-pcp - cockpit-podman - cockpit-storaged - cockpit-system - cockpit-ws A few commands to make it work: # systemctl enable --now cockpit.socket # systemctl restart cockpit.socket I accessed to https://mylocalip:9090 with a regular user. I clicked over "Limited access" to switch to administrative access. In the section of "Podman containers" a message told me about enabling the podman service or something similar, I agreed and now the LXC containers are working. I don't know how it is related... -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214845 https://bugzilla.suse.com/show_bug.cgi?id=1214845#c3 --- Comment #3 from Chris <chrisvte@gmail.com> --- Latest update applied to the AMD Ryzen 5900X machine and I still get the same behavior. # uname -a 6.4.12-1-default #1 SMP PREEMPT_DYNAMIC Fri Aug 25 08:26:31 UTC 2023 (f5aa89b) x86_64 x86_64 x86_64 GNU/Linux The AMD Ryzen 5600G machine with cockpit is working as expected, but I've been looking through the podman installation files and I didn't see anything relevant. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214845 https://bugzilla.suse.com/show_bug.cgi?id=1214845#c5 Eric van Blokland <ericvanblokland@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ericvanblokland@gmail.com --- Comment #5 from Eric van Blokland <ericvanblokland@gmail.com> --- I've ran into this is as well, but haven't had the time to look into it thoroughly. I think it is a permission issue. My current dirty workaround is: sudo systemctl stop virtlxcd sudo virtlxcd -f /etc/libvirt/virtlxcd.conf -d -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214845 https://bugzilla.suse.com/show_bug.cgi?id=1214845#c6 --- Comment #6 from Chris <chrisvte@gmail.com> --- (In reply to James Fehlig from comment #4)
What did you upgrade from? It's been a while since memory accounting was disabled. From the systemd-default-settings package:
I don't remember the previous version of the kernel, but at worst it was a month older.
See /usr/lib/systemd/system.conf.d/__20-defaults-SUSE.conf. Please check if DefaultMemoryAccounting=no. If so, provide an override to enable it.
I changed that file content from "DefaultMemoryAccounting=no" to "DefaultMemoryAccounting=yes" and now LXC is working fine. So, as a summary of what I checked: - Clean installation -> LXC not working. - Old installation -> LXC not working. - Clean installation + cockpit with Podman containers -> LXC working. - Old installation + DefaultMemoryAccounting=yes -> LXC working. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214845 https://bugzilla.suse.com/show_bug.cgi?id=1214845#c9 --- Comment #9 from Chris <chrisvte@gmail.com> --- (In reply to James Fehlig from comment #7)
Are there any cases where the memory controller is available but LXC is not working? Regardless, I think we can close this bug. Issues not related to "Unable to find 'memory' cgroups controller mount" can be handled in a new bug.
How can I check if the memory controller is available? It is only available if I leave the DefaultMemoryAccounting to yes? What about the performance penalties? -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214845 https://bugzilla.suse.com/show_bug.cgi?id=1214845#c14 --- Comment #14 from Chris <chrisvte@gmail.com> --- (In reply to James Fehlig from comment #11)
(In reply to Chris from comment #9)
How can I check if the memory controller is available? It is only available if I leave the DefaultMemoryAccounting to yes? What about the performance penalties?
For lxc, the memory controller needs to be available in /sys/fs/cgroup/machine.slice/. You can check the 'cgroup.controllers' file for controllers available at any point in the cgroup hierarchy. E.g.
# cat /sys/fs/cgroup/machine.slice/cgroup.controllers memory pids
"cat /sys/fs/cgroup/machine.slice/cgroup.controllers" is showing the same result when "DefaultMemoryAccounting" is set to "Yes" as when it is set to "No": cpu memory pids I tried what Eric said and it worked with DefaultMemoryAccounting set to "No". So what we have for now are a few workarounds to avoid the "Unable to find 'memory' cgroups controller" error and I have no clue about the root of what is wrong. How could it be marked as solved if any new or old installation has this error? -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214845 https://bugzilla.suse.com/show_bug.cgi?id=1214845#c18 --- Comment #18 from Eric van Blokland <ericvanblokland@gmail.com> --- Ok now I have better understanding of the problem... In my opinion DefaultMemoryAccounting=Yes is not and should not be required for lxc containers to run. Some change, presumable in systemd or its configuration, made this issue surface. At a first glance I think this is an lxc/systemd interoperability issue. If I read the source correctly, the libvirt LXC driver assumes it can check cgroup controller availability with its own cgroup. Under the current circumstances this assumption is wrong. When the LXC driver is run with systemd, the container process cgroup is moved under machine.slice, where the memory controller is available regardless of the DefaultMemoryAccounting setting. If the current behaviour/configuration of systemd/cgroups is correct and desired I could write a "crude" fix for the LXC driver to not check controller availability when started with systemd and the cgroupv2 backend. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214845 https://bugzilla.suse.com/show_bug.cgi?id=1214845#c19 --- Comment #19 from Chris <chrisvte@gmail.com> --- (In reply to James Fehlig from comment #15)
I suppose you have it enabled via other means. By default on TW I only see 'pids'.
I think the same. The computer which is running fine with the cockpit Podman containers activated is showing this: cpuset cpu io memory hugetlb pids rdma misc
DefaultMemoryAccounting=yes is not a workaround. It's a required config change for libvirt-lxc.
If it is required, it would be great if this config was applied automatically as part of the the virtualization tools install or written into the wiki.
Are there any new or old installations that still give "Unable to find 'memory' cgroups controller mount" error after the required config change?
No, but neither do the other two "solutions". What I would like to know is what change introduced this problem to understand it and fixed it so no one else ran into it. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214845 https://bugzilla.suse.com/show_bug.cgi?id=1214845#c25 --- Comment #25 from Eric van Blokland <ericvanblokland@gmail.com> --- (In reply to James Fehlig from comment #24)
(In reply to Eric van Blokland from comment #23)
I'm fully aware that the LXC driver is not supported by SUSE and that I have you to thank for any patches that get applied regardless. This is also why I don't mind to spend some time to fix issues and actually submit a patch.
Thanks for that!
As to the why "libvirt-lxc": I've worked with libvirt my entire professional life to run vms. So it's tech that I know (that's why libvirt). For situations where an entire vm is a bit overkill I prefer to use containers. The fact I can use the same cli and ui tools for both my vms and containers is extremely appealing.
Right, one of the main benefits of libvirt. Okay, I'll leave the lxc driver enabled with the caveat that user community is also the maintainers of the driver :-). I'll encourage bug reporters to pursue a fix upstream, at which point I'm happy to integrate in the downstream package.
Is there another libvirt container driver that would be more or less a drop in replacement for LXC? I see a few others listed, but I'm not familiar with them and apparently can't assume they're actively maintained.
lxc is the only container driver for libvirt.
Sorry for not replying earlier, I've been crazy busy. I don't mind doing some maintenance for the software I use. I will try to fix the issue in this report as soon I can, but since there are a couple of workarounds, it's not on the top of my todo list at this time. Is there anyone in particular you think I could bother if I have questions about implementation details? -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com