[Bug 1183247] New: starting lxc container failes with GDBus.Error:org.freedesktop.machine1.NoMachineForPID: PID 4370 does not belong to any known machine
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 Bug ID: 1183247 Summary: starting lxc container failes with GDBus.Error:org.freedesktop.machine1.NoMachineForPID: PID 4370 does not belong to any known machine Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: openSUSE Tumbleweed Status: NEW Severity: Normal Priority: P5 - None Component: Virtualization:Other Assignee: virt-bugs@suse.de Reporter: stefan.eisenwiener@eisenwiener.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- Created attachment 846959 --> http://bugzilla.opensuse.org/attachment.cgi?id=846959&action=edit reduced log with descriped error after Tumbleweed update to 20210307 starting an lxc container failes with GDBus.Error:org.freedesktop.machine1.NoMachineForPID: PID 4370 does not belong to any known machine. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c1 Stefan Eisenwiener <stefan.eisenwiener@eisenwiener.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P2 - High Severity|Normal |Major --- Comment #1 from Stefan Eisenwiener <stefan.eisenwiener@eisenwiener.com> --- I have made some tests with other openSUSE distributions. For testing I installed a minimal Server setup, and added libvird with lxc driver.
zypper in -y --no-recommends libvirt-daemon-lxc libvirt-client libvirt-admin libvirt-daemon-config-network libvirt-daemon-config-nwfilter
This was tested with a also minimal lxc container setup <domain type='lxc'> <name>vm1</name> <memory>500000</memory> <os> <type>exe</type> <init>/bin/sh</init> </os> <vcpu>1</vcpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/libvirt_lxc</emulator> <interface type='network'> <source network='default'/> </interface> <console type='pty' /> </devices> </domain> The results are as follows openSUSE-42.3 | WORKING openSUSE-15.1 | WORKING openSUSE-15.2 | WORKING openSUSE-tumbleweed | failed with NoMachineForPID Error I think there are some access rights missing or incorrect, but I can't find the root cause, because I'm not that firmiliar with the libvirt, and/or systemd internal workings. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c2 --- Comment #2 from Stefan Eisenwiener <stefan.eisenwiener@eisenwiener.com> --- Created attachment 847180 --> http://bugzilla.opensuse.org/attachment.cgi?id=847180&action=edit packagelist diff of snappshots 20210302 and 20210305 I have narrowed it down to the tumbleweed snapshot which begins to show the issue. tumbleweed-20210302 still working tumbleweed-20210305 shows the issue - libvirt was updated from 7.0.0 to 7.1.0 - gnutls was updated from 3.6.15 to 3.7.0 - crypto-policies has been newly added (but not completely configured yet) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c3 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CONFIRMED --- Comment #3 from James Fehlig <jfehlig@suse.com> --- (In reply to Stefan Eisenwiener from comment #2)
tumbleweed-20210302 still working tumbleweed-20210305 shows the issue - libvirt was updated from 7.0.0 to 7.1.0
Thanks for bisecting to this point! I've bisected libvirt between these releases and found commit 9c1693eff4 introduced the problem. When starting an LXC container, virLXCProcessStart() calls virCgroupNewDetectMachine(), which now calls virSystemdGetMachineUnitByPID(). The latter fails, even though the container process should be running since handshake with the lxc driver has already occurred. Investigating... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c4 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mkoutny@suse.com --- Comment #4 from James Fehlig <jfehlig@suse.com> --- It smells like a libvirt+cgroups V1 issue. I noticed my TW machine had 'systemd.legacy_systemd_cgroup_controller=0' kernel parameter, which enables the hybrid hierarchy (both V1 and V2) # mount | grep cgroup cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) I suppose the default in TW is hybrid. After removing the parameter I still have V1+V2 (and still encounter this bug) # mount | grep cgroup cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) Finally, I forced the unified (aka V2) hierarchy by adding the 'systemd.unified_cgroup_hierarchy=1' kernel parameter and now we have only V2 # mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) and the container starts successfully. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c8 Bernd Wachter <bernd-obs@wachter.fi> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bernd-obs@wachter.fi --- Comment #8 from Bernd Wachter <bernd-obs@wachter.fi> --- I've hit this same issue when testing if we can start upgrading some systems to 15.3. Libvirt version there also is 7.1.0. When forcing only v2 cgroup hierarchy I can start a container, though still have other issues making it unusable: # virsh -c lxc:// lxc-enter-namespace www1-de --noseclabel /usr/bin/bash libvirt: Cgroup error : Unable to write to '/sys/fs/cgroup/machine.slice/machine-lxc\x2d1712\x2dwww1\x2dde.scope/cgroup.procs': Device or resource busy error: internal error: Child process (27649) unexpected exit status 125 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c12 --- Comment #12 from James Fehlig <jfehlig@suse.com> --- FYI, I verified the cgroups hybrid issue still exists with libvirt.git master and filed an issue in the upstream tracker https://gitlab.com/libvirt/libvirt/-/issues/182 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c16 --- Comment #16 from OBSbugzilla Bot <bwiedemann+obsbugzillabot@suse.com> --- This is an autogenerated message for OBS integration: This bug (1183247) was mentioned in https://build.opensuse.org/request/show/924290 Factory / libvirt -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c18 Zhufa Sa <zhufa.sa@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |zhufa.sa@suse.com --- Comment #18 from Zhufa Sa <zhufa.sa@suse.com> --- Is there a plan for backport these patches to Leap 15.3? It ships Libvirt 7.1.0. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c19 --- Comment #19 from James Fehlig <jfehlig@suse.com> --- (In reply to Zhufa Sa from comment #18)
Is there a plan for backport these patches to Leap 15.3? It ships Libvirt 7.1.0.
I hadn't planned on it since the bug was filed against TW. Are you affected by this issue on 15.3? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c20 --- Comment #20 from Bernd Wachter <bernd-obs@wachter.fi> --- (In reply to James Fehlig from comment #19)
(In reply to Zhufa Sa from comment #18)
Is there a plan for backport these patches to Leap 15.3? It ships Libvirt 7.1.0.
I hadn't planned on it since the bug was filed against TW. Are you affected by this issue on 15.3?
See https://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c8 - we're stuck on Leap 15.2 for all systems hosting LXC containers until this is fixed. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c21 --- Comment #21 from Zhufa Sa <zhufa.sa@suse.com> --- (In reply to James Fehlig from comment #19)
(In reply to Zhufa Sa from comment #18)
Is there a plan for backport these patches to Leap 15.3? It ships Libvirt 7.1.0.
I hadn't planned on it since the bug was filed against TW. Are you affected by this issue on 15.3?
Yes, I affected by it. I also thinking in consideration of that the libvirtd shiped within SLE repo relevant to SLE-15-SP3, so there is a plan following ... But this is not a big problem for me because I can workaround it by using Virtualization repo from OBS, Let it go if there no any plan. Thanks. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c22 --- Comment #22 from James Fehlig <jfehlig@suse.com> --- I added the patch to the SLE15SP3/Leap15.3 libvirt package. It will be included in the next maintenance release. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c26 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED |--- --- Comment #26 from James Fehlig <jfehlig@suse.com> --- Apparently the "fix" for this issue has caused other problems and will be reverted upstream https://gitlab.com/libvirt/libvirt/-/issues/182#note_763329557 I'll wait for an upstream solution before making additional changes to the downstream libvirt packages. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c27 --- Comment #27 from Bernd Wachter <bernd-obs@wachter.fi> --- I'm now hitting this bug on Leap 15.4 on our test machine to see if we can move from 15.3 to 15.4 without issues. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c28 Eric van Blokland <ericvanblokland@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ericvanblokland@gmail.com --- Comment #28 from Eric van Blokland <ericvanblokland@gmail.com> --- I've been hit by this as well and had to shelve some upgrades. Tumbleweed is working fine at the moment. I've installed libvirtd 8.7.0 on a Leap 15.4 test machine using https://download.opensuse.org/repositories/Virtualization/15.4/Virtualizatio... but this exhibits the same behavior. So maybe it is a systemd interoperability issue? A glance through the commit history did not reveal something obviously related. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c30 --- Comment #30 from Eric van Blokland <ericvanblokland@gmail.com> --- (In reply to Michal Koutn� from comment #29)
(In reply to Eric van Blokland from comment #28)
I've been hit by this as well and had to shelve some upgrades.
Is it the same issue like the initial comment? I.e. 'GDBus.Error:org.freedesktop.machine1.NoMachineForPID:'? What process does the offending PID refer to (is it the PID 1 of the container?) Can you check what's in /proc/$PID/cgroup (when the error occurs)? TY
Tumbleweed is working fine at the moment.
FTR, openSUSE TW defaults to the unified (v2) hierarchy now and it's not affected (was confirmed in comment 4).
Figures, I thought TW was still hybrid. I'm pretty sure it's the same issue. Booting with 'systemd.unified_cgroup_hierarchy=1' resolves it. If I'm not mistaken the offending PID is that from the driver/controller(?) not the actual container. It exits too quickly to dump the cgroup information. I could try and attach a debugger if you're interested. But that would have to wait till tomorrow. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c32 --- Comment #32 from Eric van Blokland <ericvanblokland@gmail.com> --- (In reply to Michal Koutn� from comment #31)
(In reply to Eric van Blokland from comment #30)
It exits too quickly to dump the cgroup information.
Understood.
I could try and attach a debugger if you're interested. But that would have to wait till tomorrow.
Maybe the debug logs as in comment 0 would suffice. (Although they don't seem to contain the unified hierarchy path.)
As an exercise I've taken a dive into this issue. I have very little knowledge about cgroups so that has been a bit of a handicap. As mentioned in one of the comments, the lxc control process is started within the scope of the parent, libvirtd. To me it seems as if the control process is correctly added to the new scope. However, the PID remains tied to the old scope. I've tried changing the controller to use cgroup.procs instead of tasks, which does get rid of the process in the old scope. However, the last line of /proc/PID/cgroup never changes and systemd does not find the controller process in the correct unit. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c34 --- Comment #34 from Eric van Blokland <ericvanblokland@gmail.com> --- (In reply to Michal Koutn� from comment #33)
(In reply to Eric van Blokland from comment #32)
To me it seems as if the control process is correctly added to the new scope. However, the PID remains tied to the old scope.
There are multiple hierarchies, for systemd tracking purposes the only v2 is relevant.
Like I said my understanding of cgroups is extremely limited. I have no clue about the differences between v1, v2 and what unified is supposed to mean. Somehow I was led to believe that v2 is unified.
I've tried changing the controller ...
Do you refer to libvirt or lxc or other code here?
I'm referring to the code in libvirt, specifically the code for the lxc controller process (lxc_controller) that attempts to associate the process with correct cgroups
... to use cgroup.procs instead of tasks, which does get rid of the process in the old scope.
(That should only cause change if you dealt with multi-threaded processes.)
So I understood and as far as I've seen the lxc_controller process is not threaded. Imagine my confusion. Could the controller be running threads to handle IPC?
However, the last line of /proc/PID/cgroup never changes and systemd does not find the controller process in the correct unit.
The last row (assuming it's the '0::' v2 hierarchy) is the "source of truth" for systemd when it comes to unit assocition (when it runs in hybrid or unified mode).
I glanced over the systemd code and that led me to believe that if it's not using unified cgroups, it uses the controller at 1:name=systemd Or does it use the "unified code path" when v2 is available? While I am writing this I think to recall only v1 was mounted, but now I'm unsure, I would have to check. Anyway, it is clear I'm having some issues with the terminology. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c36 Eric van Blokland <ericvanblokland@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #862186|0 |1 is obsolete| | Flags|needinfo?(ericvanblokland@g | |mail.com) | --- Comment #36 from Eric van Blokland <ericvanblokland@gmail.com> --- Created attachment 862326 --> http://bugzilla.opensuse.org/attachment.cgi?id=862326&action=edit [PATCH] cgroup/LXC: Do not condition availability of v2 by controllers So if I understand correctly, in hybrid cgroups the scope needs to be set using V2 for systemd. However, libvirt/lxc treats V2 as unavailable if there are no controllers. Your patch does not work out of the box. It fails in virCgroupV2BindMount when it tries to create the directory (g_mkdir_with_parents) "/sys/fs/cgroup/unified". Which makes sense to me on a read-only file system. If I make the errors in this function non-fatal, everything seems to work as expected. If I make virCgroupBindMount do the binds in reverse order (V2 has precedence over V1) virCgroupV2BindMount succeeds without modifications. That may seem a bit dirty, but to me it's less convoluted than letting V2 be aware it should be mounted within V1. I've tested this patch with hybrid and unified cgroups, but not without systemd. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c39 --- Comment #39 from Eric van Blokland <ericvanblokland@gmail.com> --- (In reply to Michal Koutn� from comment #37)
(In reply to Eric van Blokland from comment #36)
Your patch does not work out of the box. It fails in virCgroupV2BindMount when it tries to create the directory (g_mkdir_with_parents) "/sys/fs/cgroup/unified".
Thanks for the test!
That may seem a bit dirty, but to me it's less convoluted than letting V2 be aware it should be mounted within V1.
It relies on the tmpfs mount over the ro-path in the v1 virCgroupBindMount, yeah, that's slightly dirty but nothing incorrectable (should be explained in the commit message though).
If you have any suggestions on how to improve that, let me know. I recall "do" loops are frowned upon these days, but I wanted to make explicit this loop was doing something else than the others, without messing with types. In any case I'll mention it more explicitly in the commit message.
I've tested this patch with hybrid and unified cgroups, but not without systemd.
Would you send the patch (even the current form) to the upstream [1]? (To resolve the nuances there.) I can help you if you're unfamiliar; if you send yourself, please Cc me on that mail.
You guessed correctly that I'm not familiar with mailing patches around, but I think I'll manage. I've got time tomorrow in the afternoon to tidy things up. I'll let you know if I get stuck somewhere. I'll also be sure to cc you and mention the report in the libvirt gitlab. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c40 --- Comment #40 from Eric van Blokland <ericvanblokland@gmail.com> --- The patch was accepted. I broke the clang builds in the process though. :-/ -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c41 --- Comment #41 from Eric van Blokland <ericvanblokland@gmail.com> --- Since my patch broke cgroups, it was reverted. On the mailinglist three possible fixes/workarounds were mentioned but upstream does not seem to care. I would like to see this fixed though. I would like to know which fix would be acceptable by the OpenSUSE maintainers, so I can implement that. 1. Skip obtaining the machine unit name during container startup 2. Let systemd launch the lxc controller process as a transient unit 3. Add a dummy controller to force some v2 logic -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c42 --- Comment #42 from James Fehlig <jfehlig@suse.com> --- (In reply to Eric van Blokland from comment #41)
Since my patch broke cgroups, it was reverted. On the mailinglist three possible fixes/workarounds were mentioned but upstream does not seem to care. I would like to see this fixed though. I would like to know which fix would be acceptable by the OpenSUSE maintainers, so I can implement that.
1. Skip obtaining the machine unit name during container startup 2. Let systemd launch the lxc controller process as a transient unit 3. Add a dummy controller to force some v2 logic
I'd be receptive to any fix that is isolated to the lxc driver. I'm less fond of a fix that touches common code, which could also affect the qemu driver. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c43 --- Comment #43 from Eric van Blokland <ericvanblokland@gmail.com> --- In that case it is up to you (In reply to James Fehlig from comment #42)
(In reply to Eric van Blokland from comment #41)
Since my patch broke cgroups, it was reverted. On the mailinglist three possible fixes/workarounds were mentioned but upstream does not seem to care. I would like to see this fixed though. I would like to know which fix would be acceptable by the OpenSUSE maintainers, so I can implement that.
1. Skip obtaining the machine unit name during container startup 2. Let systemd launch the lxc controller process as a transient unit 3. Add a dummy controller to force some v2 logic
I'd be receptive to any fix that is isolated to the lxc driver. I'm less fond of a fix that touches common code, which could also affect the qemu driver.
In that case it's a trade off between maintainability and reliability. In lxc_process.c there are two calls that cause issues: virLXCDomainGetMachineName lxc_process.c:1465 virCgroupNewDetectMachine lxc_process.c:1473 The purpose of these calls is to abort the container initialization in an early stage if something goes wrong. More specifically, if issues occurred with machined registration and cgroup creation. Option 1: For easy maintenance I could make a condition to only call these when the cgroup v2 backend is available or when there is no systemd at the cost of some reliability. Option 2: Alter the signature of virCgroupNewDetectMachine to have a flag to optionally check the unit name. When called from lxc_process.c we pass the flag to not check the unit name when the cgroup v2 backend is not available and systemd is present. This way we keep some reliability at the cost of (probably) more maintenance. Option 3: Implement a function with the relevant parts of virCgroupNewDetectMachine directly in lxc_process.c as a fallback for when the cgroup v2 backend is not available and systemd is present. Some reliability, more maintenance. In my opinion option 1 is sufficient. Let me know if one of these solutions would be acceptable. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c44 Eric van Blokland <ericvanblokland@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #862326|0 |1 is obsolete| | --- Comment #44 from Eric van Blokland <ericvanblokland@gmail.com> --- Created attachment 863403 --> http://bugzilla.opensuse.org/attachment.cgi?id=863403&action=edit Fix lxc container initialization with systemd and hybrid cgroups Or Option 4: In an environment with systemd and hybrid cgroups retrieve the control process child pid and perform these particular checks with it. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c45 Till D�rges <doerges@pre-sense.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |doerges@pre-sense.de --- Comment #45 from Till D�rges <doerges@pre-sense.de> --- FWIW: My host (including LXC containers) was working fine with Leap 15.3. After 'zypper dup' to 15.4 the LXC containers weren't starting anymore showing the error PID ... does not belong to any known machine Adding systemd.unified_cgroup_hierarchy=1 to the kernel cmdline didn't help. The error changed to internal error: Unable to find 'memory' cgroups controller mount Downgrading to the packages from Leap 15.3 does give me working LXC containers: # add 15.3 repos http://download.opensuse.org/distribution/leap/15.3/repo/oss/ http://download.opensuse.org/update/leap/15.3/sle/ http://download.opensuse.org/update/leap/15.3/oss # downgrade zypper install --oldpackage libvirt-daemon-driver-lxc-7.1.0-150300.6.35.2 libvirt-7.1.0-150300.6.35.2 [...] The following 52 packages are going to be downgraded: kexec-tools libvirt libvirt-client libvirt-daemon libvirt-daemon-config-network libvirt-daemon-config-nwfilter libvirt-daemon-driver-interface libvirt-daemon-driver-libxl libvirt-daemon-driver-lxc libvirt-daemon-driver-network libvirt-daemon-driver-nodedev libvirt-daemon-driver-nwfilter libvirt-daemon-driver-qemu libvirt-daemon-driver-secret libvirt-daemon-driver-storage libvirt-daemon-driver-storage-core libvirt-daemon-driver-storage-disk libvirt-daemon-driver-storage-gluster libvirt-daemon-driver-storage-iscsi libvirt-daemon-driver-storage-iscsi-direct libvirt-daemon-driver-storage-logical libvirt-daemon-driver-storage-mpath libvirt-daemon-driver-storage-rbd libvirt-daemon-driver-storage-scsi libvirt-daemon-lxc libvirt-libs lxd lxd-bash-completion python3-libvirt-python qemu qemu-audio-spice qemu-block-curl qemu-block-nfs qemu-block-rbd qemu-chardev-baum qemu-chardev-spice qemu-hw-display-qxl qemu-hw-display-virtio-gpu qemu-hw-display-virtio-gpu-pci qemu-hw-display-virtio-vga qemu-hw-s390x-virtio-gpu-ccw qemu-hw-usb-redirect qemu-hw-usb-smartcard qemu-kvm qemu-tools qemu-ui-curses qemu-ui-gtk qemu-ui-opengl qemu-ui-spice-app qemu-ui-spice-core qemu-x86 xen-libs The following 3 NEW packages are going to be installed: libbrlapi0_7 libvirglrenderer0 libvirt-bash-completion The following 3 packages are going to be REMOVED: qemu-accel-qtest qemu-accel-tcg-x86 qemu-hw-usb-host [...] # lock versions zypper addlock libvirt-daemon-driver-lxc # remove 15.3 repos -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c48 Eric van Blokland <ericvanblokland@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #863403|0 |1 is obsolete| | --- Comment #48 from Eric van Blokland <ericvanblokland@gmail.com> --- Created attachment 863686 --> http://bugzilla.opensuse.org/attachment.cgi?id=863686&action=edit Fix lxc container initialization with systemd and hybrid cgroups @James, Haha, yeah I did actually verify it worked. I'm fairly certain the forked lxc_process will consistently be the only child process at this stage of the synchronization. I can�t be 100% sure though. I've updated the patch with your suggestion. I've also added a cast on vm->pid to be consistent with other occurrences in the lxc driver. Thank you for considering my patch, happy holidays! -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c49 --- Comment #49 from Till D�rges <doerges@pre-sense.de> --- Just tested with 8.10.0-Virt.150400.1038.1 and my LXC containers are working: --- snip --- user@box:~> rpm -qa | egrep -i 'libvirt|Virt.150400' | sort libvirt-client-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-config-network-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-config-nwfilter-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-interface-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-lxc-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-network-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-nodedev-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-nwfilter-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-qemu-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-secret-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-core-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-disk-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-gluster-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-iscsi-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-iscsi-direct-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-logical-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-mpath-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-rbd-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-scsi-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-lxc-8.10.0-Virt.150400.1038.1.x86_64 libvirt-glib-1_0-0-4.0.0-150400.1.10.x86_64 libvirt-libs-8.10.0-Virt.150400.1038.1.x86_64 python3-libvirt-python-8.0.0-150400.1.6.x86_64 system-group-libvirt-20170617-150400.22.33.noarch typelib-1_0-LibvirtGLib-1_0-4.0.0-150400.1.10.x86_64 --- snap --- -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com