http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c2 --- Comment #2 from Stefan Eisenwiener --- Created attachment 847180 --> http://bugzilla.opensuse.org/attachment.cgi?id=847180&action=edit packagelist diff of snappshots 20210302 and 20210305 I have narrowed it down to the tumbleweed snapshot which begins to show the issue. tumbleweed-20210302 still working tumbleweed-20210305 shows the issue - libvirt was updated from 7.0.0 to 7.1.0 - gnutls was updated from 3.6.15 to 3.7.0 - crypto-policies has been newly added (but not completely configured yet) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c4 James Fehlig changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mkoutny@suse.com --- Comment #4 from James Fehlig --- It smells like a libvirt+cgroups V1 issue. I noticed my TW machine had 'systemd.legacy_systemd_cgroup_controller=0' kernel parameter, which enables the hybrid hierarchy (both V1 and V2) # mount | grep cgroup cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) I suppose the default in TW is hybrid. After removing the parameter I still have V1+V2 (and still encounter this bug) # mount | grep cgroup cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) Finally, I forced the unified (aka V2) hierarchy by adding the 'systemd.unified_cgroup_hierarchy=1' kernel parameter and now we have only V2 # mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) and the container starts successfully. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c12 --- Comment #12 from James Fehlig --- FYI, I verified the cgroups hybrid issue still exists with libvirt.git master and filed an issue in the upstream tracker https://gitlab.com/libvirt/libvirt/-/issues/182 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c18 Zhufa Sa changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |zhufa.sa@suse.com --- Comment #18 from Zhufa Sa --- Is there a plan for backport these patches to Leap 15.3? It ships Libvirt 7.1.0. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c20 --- Comment #20 from Bernd Wachter --- (In reply to James Fehlig from comment #19)

(In reply to Zhufa Sa from comment #18)

...

Is there a plan for backport these patches to Leap 15.3? It ships Libvirt 7.1.0.

I hadn't planned on it since the bug was filed against TW. Are you affected by this issue on 15.3?

See https://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c8 - we're stuck on Leap 15.2 for all systems hosting LXC containers until this is fixed. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c22 --- Comment #22 from James Fehlig --- I added the patch to the SLE15SP3/Leap15.3 libvirt package. It will be included in the next maintenance release. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c27 --- Comment #27 from Bernd Wachter --- I'm now hitting this bug on Leap 15.4 on our test machine to see if we can move from 15.3 to 15.4 without issues. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c30 --- Comment #30 from Eric van Blokland --- (In reply to Michal Koutn� from comment #29)

(In reply to Eric van Blokland from comment #28)

...

I've been hit by this as well and had to shelve some upgrades.

Is it the same issue like the initial comment? I.e. 'GDBus.Error:org.freedesktop.machine1.NoMachineForPID:'? What process does the offending PID refer to (is it the PID 1 of the container?) Can you check what's in /proc/$PID/cgroup (when the error occurs)? TY

...

Tumbleweed is working fine at the moment.

FTR, openSUSE TW defaults to the unified (v2) hierarchy now and it's not affected (was confirmed in comment 4).

Figures, I thought TW was still hybrid. I'm pretty sure it's the same issue. Booting with 'systemd.unified_cgroup_hierarchy=1' resolves it. If I'm not mistaken the offending PID is that from the driver/controller(?) not the actual container. It exits too quickly to dump the cgroup information. I could try and attach a debugger if you're interested. But that would have to wait till tomorrow. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c34 --- Comment #34 from Eric van Blokland --- (In reply to Michal Koutn� from comment #33)

(In reply to Eric van Blokland from comment #32)

...

To me it seems as if the control process is correctly added to the new scope. However, the PID remains tied to the old scope.

There are multiple hierarchies, for systemd tracking purposes the only v2 is relevant.

Like I said my understanding of cgroups is extremely limited. I have no clue about the differences between v1, v2 and what unified is supposed to mean. Somehow I was led to believe that v2 is unified.

...

I've tried changing the controller ...

Do you refer to libvirt or lxc or other code here?

I'm referring to the code in libvirt, specifically the code for the lxc controller process (lxc_controller) that attempts to associate the process with correct cgroups

...

... to use cgroup.procs instead of tasks, which does get rid of the process in the old scope.

(That should only cause change if you dealt with multi-threaded processes.)

So I understood and as far as I've seen the lxc_controller process is not threaded. Imagine my confusion. Could the controller be running threads to handle IPC?

...

However, the last line of /proc/PID/cgroup never changes and systemd does not find the controller process in the correct unit.

The last row (assuming it's the '0::' v2 hierarchy) is the "source of truth" for systemd when it comes to unit assocition (when it runs in hybrid or unified mode).

I glanced over the systemd code and that led me to believe that if it's not using unified cgroups, it uses the controller at 1:name=systemd Or does it use the "unified code path" when v2 is available? While I am writing this I think to recall only v1 was mounted, but now I'm unsure, I would have to check. Anyway, it is clear I'm having some issues with the terminology. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c39 --- Comment #39 from Eric van Blokland --- (In reply to Michal Koutn� from comment #37)

(In reply to Eric van Blokland from comment #36)

...

Your patch does not work out of the box. It fails in virCgroupV2BindMount when it tries to create the directory (g_mkdir_with_parents) "/sys/fs/cgroup/unified".

Thanks for the test!

...

That may seem a bit dirty, but to me it's less convoluted than letting V2 be aware it should be mounted within V1.

It relies on the tmpfs mount over the ro-path in the v1 virCgroupBindMount, yeah, that's slightly dirty but nothing incorrectable (should be explained in the commit message though).

If you have any suggestions on how to improve that, let me know. I recall "do" loops are frowned upon these days, but I wanted to make explicit this loop was doing something else than the others, without messing with types. In any case I'll mention it more explicitly in the commit message.

...

I've tested this patch with hybrid and unified cgroups, but not without systemd.

Would you send the patch (even the current form) to the upstream [1]? (To resolve the nuances there.) I can help you if you're unfamiliar; if you send yourself, please Cc me on that mail.

[1] https://libvirt.org/submitting-patches.html

You guessed correctly that I'm not familiar with mailing patches around, but I think I'll manage. I've got time tomorrow in the afternoon to tidy things up. I'll let you know if I get stuck somewhere. I'll also be sure to cc you and mention the report in the libvirt gitlab. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c41 --- Comment #41 from Eric van Blokland --- Since my patch broke cgroups, it was reverted. On the mailinglist three possible fixes/workarounds were mentioned but upstream does not seem to care. I would like to see this fixed though. I would like to know which fix would be acceptable by the OpenSUSE maintainers, so I can implement that. 1. Skip obtaining the machine unit name during container startup 2. Let systemd launch the lxc controller process as a transient unit 3. Add a dummy controller to force some v2 logic -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c43 --- Comment #43 from Eric van Blokland --- In that case it is up to you (In reply to James Fehlig from comment #42)

(In reply to Eric van Blokland from comment #41)

...

Since my patch broke cgroups, it was reverted. On the mailinglist three possible fixes/workarounds were mentioned but upstream does not seem to care. I would like to see this fixed though. I would like to know which fix would be acceptable by the OpenSUSE maintainers, so I can implement that.

1. Skip obtaining the machine unit name during container startup 2. Let systemd launch the lxc controller process as a transient unit 3. Add a dummy controller to force some v2 logic

I'd be receptive to any fix that is isolated to the lxc driver. I'm less fond of a fix that touches common code, which could also affect the qemu driver.

In that case it's a trade off between maintainability and reliability. In lxc_process.c there are two calls that cause issues: virLXCDomainGetMachineName lxc_process.c:1465 virCgroupNewDetectMachine lxc_process.c:1473 The purpose of these calls is to abort the container initialization in an early stage if something goes wrong. More specifically, if issues occurred with machined registration and cgroup creation. Option 1: For easy maintenance I could make a condition to only call these when the cgroup v2 backend is available or when there is no systemd at the cost of some reliability. Option 2: Alter the signature of virCgroupNewDetectMachine to have a flag to optionally check the unit name. When called from lxc_process.c we pass the flag to not check the unit name when the cgroup v2 backend is not available and systemd is present. This way we keep some reliability at the cost of (probably) more maintenance. Option 3: Implement a function with the relevant parts of virCgroupNewDetectMachine directly in lxc_process.c as a fallback for when the cgroup v2 backend is not available and systemd is present. Some reliability, more maintenance. In my opinion option 1 is sufficient. Let me know if one of these solutions would be acceptable. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c45 Till D�rges changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |doerges@pre-sense.de --- Comment #45 from Till D�rges --- FWIW: My host (including LXC containers) was working fine with Leap 15.3. After 'zypper dup' to 15.4 the LXC containers weren't starting anymore showing the error PID ... does not belong to any known machine Adding systemd.unified_cgroup_hierarchy=1 to the kernel cmdline didn't help. The error changed to internal error: Unable to find 'memory' cgroups controller mount Downgrading to the packages from Leap 15.3 does give me working LXC containers: # add 15.3 repos http://download.opensuse.org/distribution/leap/15.3/repo/oss/ http://download.opensuse.org/update/leap/15.3/sle/ http://download.opensuse.org/update/leap/15.3/oss # downgrade zypper install --oldpackage libvirt-daemon-driver-lxc-7.1.0-150300.6.35.2 libvirt-7.1.0-150300.6.35.2 [...] The following 52 packages are going to be downgraded: kexec-tools libvirt libvirt-client libvirt-daemon libvirt-daemon-config-network libvirt-daemon-config-nwfilter libvirt-daemon-driver-interface libvirt-daemon-driver-libxl libvirt-daemon-driver-lxc libvirt-daemon-driver-network libvirt-daemon-driver-nodedev libvirt-daemon-driver-nwfilter libvirt-daemon-driver-qemu libvirt-daemon-driver-secret libvirt-daemon-driver-storage libvirt-daemon-driver-storage-core libvirt-daemon-driver-storage-disk libvirt-daemon-driver-storage-gluster libvirt-daemon-driver-storage-iscsi libvirt-daemon-driver-storage-iscsi-direct libvirt-daemon-driver-storage-logical libvirt-daemon-driver-storage-mpath libvirt-daemon-driver-storage-rbd libvirt-daemon-driver-storage-scsi libvirt-daemon-lxc libvirt-libs lxd lxd-bash-completion python3-libvirt-python qemu qemu-audio-spice qemu-block-curl qemu-block-nfs qemu-block-rbd qemu-chardev-baum qemu-chardev-spice qemu-hw-display-qxl qemu-hw-display-virtio-gpu qemu-hw-display-virtio-gpu-pci qemu-hw-display-virtio-vga qemu-hw-s390x-virtio-gpu-ccw qemu-hw-usb-redirect qemu-hw-usb-smartcard qemu-kvm qemu-tools qemu-ui-curses qemu-ui-gtk qemu-ui-opengl qemu-ui-spice-app qemu-ui-spice-core qemu-x86 xen-libs The following 3 NEW packages are going to be installed: libbrlapi0_7 libvirglrenderer0 libvirt-bash-completion The following 3 packages are going to be REMOVED: qemu-accel-qtest qemu-accel-tcg-x86 qemu-hw-usb-host [...] # lock versions zypper addlock libvirt-daemon-driver-lxc # remove 15.3 repos -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1183247 http://bugzilla.opensuse.org/show_bug.cgi?id=1183247#c49 --- Comment #49 from Till D�rges --- Just tested with 8.10.0-Virt.150400.1038.1 and my LXC containers are working: --- snip --- user@box:~> rpm -qa | egrep -i 'libvirt|Virt.150400' | sort libvirt-client-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-config-network-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-config-nwfilter-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-interface-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-lxc-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-network-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-nodedev-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-nwfilter-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-qemu-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-secret-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-core-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-disk-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-gluster-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-iscsi-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-iscsi-direct-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-logical-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-mpath-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-rbd-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-driver-storage-scsi-8.10.0-Virt.150400.1038.1.x86_64 libvirt-daemon-lxc-8.10.0-Virt.150400.1038.1.x86_64 libvirt-glib-1_0-0-4.0.0-150400.1.10.x86_64 libvirt-libs-8.10.0-Virt.150400.1038.1.x86_64 python3-libvirt-python-8.0.0-150400.1.6.x86_64 system-group-libvirt-20170617-150400.22.33.noarch typelib-1_0-LibvirtGLib-1_0-4.0.0-150400.1.10.x86_64 --- snap --- -- You are receiving this mail because: You are on the CC list for the bug.