[Bug 1202191] New: Since 20220804 virt manager / qemu no longer works with apparmor. VM's fail to launch.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 Bug ID: 1202191 Summary: Since 20220804 virt manager / qemu no longer works with apparmor. VM's fail to launch. Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: openSUSE Tumbleweed Status: NEW Severity: Normal Priority: P5 - None Component: KVM Assignee: kvm-bugs@suse.de Reporter: andy_millman@yahoo.co.uk QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0 Build Identifier: When apparmor is enabled in /etc/libvirt/qemu.conf, all VMs fail to launch in virt-manager and instead give the error: Error starting domain: internal error: cannot load AppArmor profile 'libvirt-4f42e50f-0068-47b6-b172-04812e0cb040' Reproducible: Always Steps to Reproduce: 1.Make sure you are on Tumbleweed snapshot 20220804 2.Make sure security_driver = "apparmor" is set in /etc/libvirt/qemu.conf, or at least that it is not set to "none" or "selinux". 3.Launch virt-manager and attempt to start a VM Actual Results: Doesn't launch VM and instead prints out error message: Error starting domain: internal error: cannot load AppArmor profile 'libvirt-4f42e50f-0068-47b6-b172-04812e0cb040' Expected Results: Should launch the VM without error -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c1 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jfehlig@suse.com --- Comment #1 from James Fehlig <jfehlig@suse.com> --- (In reply to Andy Millman from comment #0)
Steps to Reproduce: 1.Make sure you are on Tumbleweed snapshot 20220804
Is this a recent regression? Did it work on previous snapshots? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c2 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |andy_millman@yahoo.co.uk Flags| |needinfo?(andy_millman@yaho | |o.co.uk) --- Comment #2 from James Fehlig <jfehlig@suse.com> --- (In reply to Andy Millman from comment #0)
Reproducible: Always
Steps to Reproduce: 1.Make sure you are on Tumbleweed snapshot 20220804 2.Make sure security_driver = "apparmor" is set in /etc/libvirt/qemu.conf, or at least that it is not set to "none" or "selinux".
Do you also have 'security_default_confined = 1' set?
3.Launch virt-manager and attempt to start a VM Actual Results: Doesn't launch VM and instead prints out error message:
Error starting domain: internal error: cannot load AppArmor profile 'libvirt-4f42e50f-0068-47b6-b172-04812e0cb040'
If possible, enable debug in /etc/libvirt/libvirtd.conf and provide the (verbose!) output when attempting to start the VM. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c3 --- Comment #3 from Andy Millman <andy_millman@yahoo.co.uk> --- (In reply to James Fehlig from comment #1)
Is this a recent regression? Did it work on previous snapshots?
Yes it is a recent regression. Worked fine in all snapshots prior to 20220804.
Do you also have 'security_default_confined = 1' set?
Yes I have that option set
If possible, enable debug in /etc/libvirt/libvirtd.conf and provide the (verbose!) output when attempting to start the VM.
I'll see if I can get some system downtime so I can try this. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c4 Andy Millman <andy_millman@yahoo.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(andy_millman@yaho | |o.co.uk) | --- Comment #4 from Andy Millman <andy_millman@yahoo.co.uk> --- (In reply to James Fehlig from comment #2)
Error starting domain: internal error: cannot load AppArmor profile 'libvirt-4f42e50f-0068-47b6-b172-04812e0cb040'
If possible, enable debug in /etc/libvirt/libvirtd.conf and provide the (verbose!) output when attempting to start the VM.
I could only do log-level 4 (errors), otherwise output was 300k lines. Hopefully this is sufficient: 2022-08-09 14:12:25.640+0000: 15791: info : libvirt version: 8.6.0 2022-08-09 14:12:25.640+0000: 15791: error : virCommandWait:2697 : internal error: Child process (LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/virt-aa-helper -r -u libvirt-68d02321-8c35-4ef4-8dbe-61080fd3794f) unexpected exit status 126: libvirt: error : cannot execute binary /usr/libexec/virt-aa-helper: Permission denied 2022-08-09 14:12:25.640+0000: 15791: error : AppArmorGenSecurityLabel:428 : internal error: cannot load AppArmor profile 'libvirt-68d02321-8c35-4ef4-8dbe-61080fd3794f' -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c5 --- Comment #5 from James Fehlig <jfehlig@suse.com> --- (In reply to Andy Millman from comment #4)
Hopefully this is sufficient:
I think so.
2022-08-09 14:12:25.640+0000: 15791: info : libvirt version: 8.6.0 2022-08-09 14:12:25.640+0000: 15791: error : virCommandWait:2697 : internal error: Child process (LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/virt-aa-helper -r -u libvirt-68d02321-8c35-4ef4-8dbe-61080fd3794f) unexpected exit status 126: libvirt: error : cannot execute binary /usr/libexec/virt-aa-helper: Permission denied
For the longest time, libvirt redefined libexecdir to /usr/lib64/libvirt. I recently removed that redefinition, in which case the distro-defined /usr/libexec is used. But odd thing is I cant reproduce this bug. I had even tested confined VMs before submitting the change to Factory, to ensure all profiles could cope with moving things like iohelper, leasehelper, etc to /usr/libexec. I suppose apparmor dumps something to /var/log/audit/audit.log when the error occurs. Can you provide that? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c6 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|kvm-bugs@suse.de |jfehlig@suse.com --- Comment #6 from James Fehlig <jfehlig@suse.com> --- Forgot to reassign to myself when making the last comment... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c7 --- Comment #7 from Andy Millman <andy_millman@yahoo.co.uk> --- (In reply to James Fehlig from comment #5)
But odd thing is I cant reproduce this bug. I had even tested confined VMs before submitting the change to Factory, to ensure all profiles could cope with moving things like iohelper, leasehelper, etc to /usr/libexec.
Weird. One other person on reddit has reported this bug, so I know it's not just me. But on the other hand I would have expected more people to have reported this, so possibly only affecting some users?
I suppose apparmor dumps something to /var/log/audit/audit.log when the error occurs. Can you provide that?
/var/log/audit/audit.log contains lines of the form: type=AVC msg=audit(1660069242.885:1229): apparmor="DENIED" operation="file_mmap" profile="dnsmasq//libvirt_leaseshelper" name="/usr/libexec/libvirt_leaseshelper" pid=7328 comm="libvirt_leasesh" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 type=ANOM_ABEND msg=audit(1660069242.885:1230): auid=4294967295 uid=0 gid=0 ses=4294967295 subj==dnsmasq//libvirt_leaseshelper (enforce) pid=7328 comm="libvirt_leasesh" exe="/usr/libexec/libvirt_leaseshelper" sig=11 res=1 Hope that helps! -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c8 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |suse-beta@cboltz.de --- Comment #8 from James Fehlig <jfehlig@suse.com> --- (In reply to Andy Millman from comment #7)
Weird. One other person on reddit has reported this bug, so I know it's not just me. But on the other hand I would have expected more people to have reported this, so possibly only affecting some users?
I suppose. Let's try to figure out what is different between our setups. My TW is a minimal installation via autoyast. I have the following apparmor packages installed libapparmor1-3.0.6-1.1.x86_64 apparmor-abstractions-3.0.6-1.1.noarch apparmor-parser-3.0.6-1.1.x86_64 apparmor-profiles-3.0.6-2.1.noarch And with VM confinement enabled in /etc/libvirt/qemu.conf and one VM running, the output of apparmor_status apparmor module is loaded. 61 profiles are loaded. 61 profiles are in enforce mode. /usr/bin/lessopen.sh apache2 apache2//DEFAULT_URI apache2//HANDLING_UNTRUSTED_INPUT apache2//phpsysinfo avahi-daemon dnsmasq dnsmasq//libvirt_leaseshelper dovecot dovecot-anvil dovecot-auth dovecot-config dovecot-deliver dovecot-dict dovecot-dovecot-auth dovecot-dovecot-lda dovecot-dovecot-lda//sendmail dovecot-imap dovecot-imap-login dovecot-lmtp dovecot-log dovecot-managesieve dovecot-managesieve-login dovecot-pop3 dovecot-pop3-login dovecot-script-login dovecot-ssl-params dovecot-stats identd klogd libvirt-f25e648e-1e3e-4316-8702-ae3cbf6aded0 libvirtd libvirtd//qemu_bridge_helper lsb_release mdnsd nmbd nscd ntpd nvidia_modprobe nvidia_modprobe//kmod php-fpm ping samba-bgqd samba-dcerpcd samba-rpcd samba-rpcd-classic samba-rpcd-spoolss smbd smbldap-useradd smbldap-useradd///etc/init.d/nscd syslog-ng syslogd traceroute virt-aa-helper virtqemud virtqemud//qemu_bridge_helper virtxend winbindd zgrep zgrep//helper zgrep//sed 0 profiles are in complain mode. 0 profiles are in kill mode. 0 profiles are in unconfined mode. 2 processes have profiles defined. 2 processes are in enforce mode. /usr/bin/qemu-system-x86_64 (2258) libvirt-f25e648e-1e3e-4316-8702-ae3cbf6aded0 /usr/sbin/libvirtd (2182) libvirtd 0 processes are in complain mode. 0 processes are unconfined but have a profile defined. 0 processes are in mixed mode. 0 processes are in kill mode.
type=AVC msg=audit(1660069242.885:1229): apparmor="DENIED" operation="file_mmap" profile="dnsmasq//libvirt_leaseshelper" name="/usr/libexec/libvirt_leaseshelper" pid=7328 comm="libvirt_leasesh" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 type=ANOM_ABEND msg=audit(1660069242.885:1230): auid=4294967295 uid=0 gid=0 ses=4294967295 subj==dnsmasq//libvirt_leaseshelper (enforce) pid=7328 comm="libvirt_leasesh" exe="/usr/libexec/libvirt_leaseshelper" sig=11 res=1
I think we need help from an apparmor maintainer to properly decipher these messages. But they should be unrelated to your problem of "error : cannot execute binary /usr/libexec/virt-aa-helper: Permission denied". FTR # ll /usr/libexec/virt-aa-helper -rwxr-xr-x 1 root root 39616 Aug 4 06:21 /usr/libexec/virt-aa-helper -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c9 --- Comment #9 from Christian Boltz <suse-beta@cboltz.de> --- (In reply to James Fehlig from comment #8)
(In reply to Andy Millman from comment #7)
type=AVC msg=audit(1660069242.885:1229): apparmor="DENIED" operation="file_mmap" profile="dnsmasq//libvirt_leaseshelper" name="/usr/libexec/libvirt_leaseshelper" pid=7328 comm="libvirt_leasesh" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
This is bug 1202161 - and already fixed (IIRC it's already in the latest Tumbleweed snapshot).
type=ANOM_ABEND msg=audit(1660069242.885:1230): auid=4294967295 uid=0 gid=0 ses=4294967295 subj==dnsmasq//libvirt_leaseshelper (enforce) pid=7328 comm="libvirt_leasesh" exe="/usr/libexec/libvirt_leaseshelper" sig=11 res=1
That's a follow-up problem of the previous denial.
I think we need help from an apparmor maintainer to properly decipher these messages. But they should be unrelated to your problem of "error : cannot execute binary /usr/libexec/virt-aa-helper: Permission denied".
Most boring question first: Do you have *.rpmnew files in /etc/apparmor.d/ (especially for dnsmasq or *virt*)? If so, please rename them (remove the '.rpmnew' part, overwriting the old existing profile - doing a diff first never hurts). If you still see problems, please check your audit.log for more DENIED lines. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c10 --- Comment #10 from Andy Millman <andy_millman@yahoo.co.uk> --- (In reply to Christian Boltz from comment #9)
Most boring question first: Do you have *.rpmnew files in /etc/apparmor.d/ (especially for dnsmasq or *virt*)? If so, please rename them (remove the '.rpmnew' part, overwriting the old existing profile - doing a diff first never hurts).
Ok thanks! Yes I had a usr.sbin.libvirtd.rpmnew from 6th August in the /etc/apparmor.d folder which I've just renamed to usr.sbin.libvirtd
If you still see problems, please check your audit.log for more DENIED lines.
I rebooted and sadly not working yet. When trying to start a VM with apparmor enabled I get the following: type=AVC msg=audit(1660224118.236:305): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/virt-aa-helper" pid=15720 comm="rpc-libvirtd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 type=VIRT_RESOURCE msg=audit(1660224118.252:306): pid=12114 uid=0 auid=4294967295 ses=4294967295 subj==libvirtd (enforce) msg='virt=kvm resrc=disk reason=start vm="UEFI" uuid=68d02321-8c35-4ef4-8dbe-61080fd3794f old-disk="?" new-disk="/run/media/kmk/Virtual/libvirt/UEFI.qcow2" exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' type=VIRT_RESOURCE msg=audit(1660224118.252:307): pid=12114 uid=0 auid=4294967295 ses=4294967295 subj==libvirtd (enforce) msg='virt=kvm resrc=net reason=start vm="UEFI" uuid=68d02321-8c35-4ef4-8dbe-61080fd3794f old-net="?" new-net="52:54:00:61:79:87" exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' type=VIRT_RESOURCE msg=audit(1660224118.252:308): pid=12114 uid=0 auid=4294967295 ses=4294967295 subj==libvirtd (enforce) msg='virt=kvm resrc=dev reason=start vm="UEFI" uuid=68d02321-8c35-4ef4-8dbe-61080fd3794f bus=usb device=555342207265646972646576 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' type=VIRT_RESOURCE msg=audit(1660224118.252:309): pid=12114 uid=0 auid=4294967295 ses=4294967295 subj==libvirtd (enforce) msg='virt=kvm resrc=dev reason=start vm="UEFI" uuid=68d02321-8c35-4ef4-8dbe-61080fd3794f bus=usb device=555342207265646972646576 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' type=VIRT_RESOURCE msg=audit(1660224118.252:310): pid=12114 uid=0 auid=4294967295 ses=4294967295 subj==libvirtd (enforce) msg='virt=kvm resrc=rng reason=start vm="UEFI" uuid=68d02321-8c35-4ef4-8dbe-61080fd3794f old-rng="?" new-rng="/dev/urandom" exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' type=VIRT_RESOURCE msg=audit(1660224118.252:311): pid=12114 uid=0 auid=4294967295 ses=4294967295 subj==libvirtd (enforce) msg='virt=kvm resrc=mem reason=start vm="UEFI" uuid=68d02321-8c35-4ef4-8dbe-61080fd3794f old-mem=0 new-mem=8388608 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' type=VIRT_RESOURCE msg=audit(1660224118.252:312): pid=12114 uid=0 auid=4294967295 ses=4294967295 subj==libvirtd (enforce) msg='virt=kvm resrc=vcpu reason=start vm="UEFI" uuid=68d02321-8c35-4ef4-8dbe-61080fd3794f old-vcpu=0 new-vcpu=2 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=success' type=VIRT_CONTROL msg=audit(1660224118.252:313): pid=12114 uid=0 auid=4294967295 ses=4294967295 subj==libvirtd (enforce) msg='virt=kvm op=start reason=booted vm="UEFI" uuid=68d02321-8c35-4ef4-8dbe-61080fd3794f vm-pid=0 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed' -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c11 --- Comment #11 from Christian Boltz <suse-beta@cboltz.de> --- (In reply to Andy Millman from comment #10)
I rebooted and sadly not working yet. When trying to start a VM with apparmor enabled I get the following:
type=AVC msg=audit(1660224118.236:305): apparmor="DENIED" operation="exec" profile="libvirtd" name="/usr/libexec/virt-aa-helper" pid=15720 comm="rpc-libvirtd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
We are making progress - that's a new error/denial ;-) The libvirtd profile contains /usr/libexec/* PUxr, which should allow to execute everything in /usr/libexec/ (even if no profile exists for it, in this case it will run unconfined). Please check if your /etc/apparmor.d/usr.sbin.libvirtd really includes that rule (should be in line 109). (IMHO that rule is too broad and insecure given the large amount of binaries in /usr/libexec/, but that's another topic.) Please also show the output of ls -l /etc/apparmor.d/usr*virt* /var/cache/apparmor/*/usr*virt* Wild guess: if your (renamed) usr.sbin.libvirtd kept the timestamp from the rpm, your profile cache might still have a cache file of the previous profile. The above "ls -l" will show that. You can try touch /etc/apparmor.d/usr.sbin.libvirtd ; rcapparmor reload to ensure the cache gets updated - but please do that only _after_ saving the "ls -l" output. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1202191 http://bugzilla.opensuse.org/show_bug.cgi?id=1202191#c12 Andy Millman <andy_millman@yahoo.co.uk> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #12 from Andy Millman <andy_millman@yahoo.co.uk> --- (In reply to Christian Boltz from comment #11)
The libvirtd profile contains /usr/libexec/* PUxr, which should allow to execute everything in /usr/libexec/ (even if no profile exists for it, in this case it will run unconfined).
Please check if your /etc/apparmor.d/usr.sbin.libvirtd really includes that rule (should be in line 109).
Yes it includes that line.
(IMHO that rule is too broad and insecure given the large amount of binaries in /usr/libexec/, but that's another topic.)
Please also show the output of ls -l /etc/apparmor.d/usr*virt* /var/cache/apparmor/*/usr*virt*
Wild guess: if your (renamed) usr.sbin.libvirtd kept the timestamp from the rpm, your profile cache might still have a cache file of the previous profile. The above "ls -l" will show that. You can try touch /etc/apparmor.d/usr.sbin.libvirtd ; rcapparmor reload to ensure the cache gets updated - but please do that only _after_ saving the "ls -l" output.
Ok you guessed the correct issue! It was cacheing an old copy of the file. The touch command fixed everything. Sincere thanks for your help in tracking down and helping to fix this issue. Much appreciated! Also sincere thanks to James Fehlig for all the help. Much appreciated! -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com