[Bug 1125841] New: AppArmor security driver not enabled for QEMU in libvirt
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 Bug ID: 1125841 Summary: AppArmor security driver not enabled for QEMU in libvirt Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: openSUSE Factory Status: NEW Severity: Normal Priority: P5 - None Component: Virtualization:Tools Assignee: virt-bugs@suse.de Reporter: supercoolemail@seznam.cz QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36 Build Identifier: Setting security_driver = "apparmor" in /etc/libvirt/qemu.conf or enabling apparmor individually per domain leads to error with libvirt 5: Security driver apparmor not enabled Reproducible: Always Steps to Reproduce: 1. Set apparmor as security driver in /etc/libvirt/qemu.conf . 2. Use libvirt to start QEMU domain. Actual Results: Error: Security driver apparmor not enabled Failed to initialize security drivers Initialization of QEMU state driver failed: internal error: Failed to initialize security drivers Expected Results: Domain would start and be confined by AppArmor as it used to be before libvirt 5. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 Martin Kalivoda <supercoolemail@seznam.cz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |supercoolemail@seznam.cz -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c1 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(supercoolemail@se | |znam.cz) --- Comment #1 from James Fehlig <jfehlig@suse.com> --- Is apparmor running? E.g. what is the output of 'systemctl status apparmor.service' and 'aa-status'? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c2 --- Comment #2 from Martin Kalivoda <supercoolemail@seznam.cz> --- (In reply to James Fehlig from comment #1)
Is apparmor running? E.g. what is the output of 'systemctl status apparmor.service' and 'aa-status'?
● apparmor.service - Load AppArmor profiles Loaded: loaded (/usr/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled) Active: active (exited) since Tue 2019-02-19 01:07:46 CET; 23h ago Main PID: 696 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 4915) Memory: 0B CGroup: /system.slice/apparmor.service Feb 19 01:07:38 vhost4 systemd[1]: Starting Load AppArmor profiles... Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Restarting AppArmor Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Reloading AppArmor profiles Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Skipped: /etc/apparmor.d/libvirt Feb 19 01:07:46 vhost4 systemd[1]: Started Load AppArmor profiles. ------------- apparmor module is loaded. 51 profiles are loaded. 50 profiles are in enforce mode. (omitted) 1 profiles are in complain mode. /usr/lib/systemd/system-generators/lvm2-activation-generator 8 processes have profiles defined. 8 processes are in enforce mode. (omitted) 0 processes are in complain mode. 0 processes are unconfined but have a profile defined. No mentions of libvirt in any of the redacted parts -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c3 --- Comment #3 from Martin Kalivoda <supercoolemail@seznam.cz> --- Well, there are of course libvirtd profiles (libvirtd, libvirtd//qemu_bridge_helper, virt-aa-helper), but not domain profiles (I have previously working profiles in /etc/apparmor.d/libvirt). -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c4 --- Comment #4 from Martin Kalivoda <supercoolemail@seznam.cz> --- It should be noted, that this is quite worrying me for another reason: comments in /etc/libvirt/qemu.conf say that # SUSE Note: # Currently, Apparmor is the default security framework in SUSE # distros. If Apparmor is enabled on the host, libvirtd is # generously confined but users must opt-in to confine qemu # instances. Change this to a non-zero value to enable default # Apparmor confinement of qemu instances. However, changing the value of security_default_confined to 1 will NOT trigger the error and it will leave everything unconfined. Only reason why I got hit by this is because I am cautious about default and changed the security_driver explicitly. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c5 --- Comment #5 from James Fehlig <jfehlig@suse.com> --- (In reply to Martin Kalivoda from comment #2)
(In reply to James Fehlig from comment #1)
Is apparmor running? E.g. what is the output of 'systemctl status apparmor.service' and 'aa-status'?
● apparmor.service - Load AppArmor profiles Loaded: loaded (/usr/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled) Active: active (exited) since Tue 2019-02-19 01:07:46 CET; 23h ago Main PID: 696 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 4915) Memory: 0B CGroup: /system.slice/apparmor.service
Feb 19 01:07:38 vhost4 systemd[1]: Starting Load AppArmor profiles... Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Restarting AppArmor Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Reloading AppArmor profiles Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Skipped: /etc/apparmor.d/libvirt
Why was this skipped? Do you get any errors parsing the libvirtd profile? E.g. try 'apparmor_parser -r /etc/apparmor.d/usr.sbin.libvirtd' and see if any parsing errors are reported.
Feb 19 01:07:46 vhost4 systemd[1]: Started Load AppArmor profiles.
-------------
apparmor module is loaded. 51 profiles are loaded. 50 profiles are in enforce mode. (omitted) 1 profiles are in complain mode. /usr/lib/systemd/system-generators/lvm2-activation-generator 8 processes have profiles defined. 8 processes are in enforce mode. (omitted) 0 processes are in complain mode. 0 processes are unconfined but have a profile defined.
No mentions of libvirt in any of the redacted parts
/usr/sbin/libvirtd is not mentioned in the output of 'aa-status'? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c6 --- Comment #6 from James Fehlig <jfehlig@suse.com> --- (In reply to Martin Kalivoda from comment #3)
Well, there are of course libvirtd profiles (libvirtd, libvirtd//qemu_bridge_helper, virt-aa-helper), but not domain profiles (I have previously working profiles in /etc/apparmor.d/libvirt).
Domain profiles in /etc/apparmor.d/libvirt/ are generated when starting a VM and removed when shutting it down. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c7 --- Comment #7 from James Fehlig <jfehlig@suse.com> --- (In reply to Martin Kalivoda from comment #4)
It should be noted, that this is quite worrying me for another reason: comments in /etc/libvirt/qemu.conf say that # SUSE Note: # Currently, Apparmor is the default security framework in SUSE # distros. If Apparmor is enabled on the host, libvirtd is # generously confined but users must opt-in to confine qemu # instances. Change this to a non-zero value to enable default # Apparmor confinement of qemu instances. However, changing the value of security_default_confined to 1 will NOT trigger the error and it will leave everything unconfined. Only reason why I got hit by this is because I am cautious about default and changed the security_driver explicitly.
Since the libvirt apparmor driver is not loaded, the default security model is 'none'. Setting security_default_confined has no effect when the active model is a no-op. -- You are receiving this mail because: You are on the CC list for the bug.
Domain profiles in /etc/apparmor.d/libvirt/ are generated when starting a VM and removed when shutting it down. This does not seems right, because https://gitlab.com/apparmor/apparmor/wikis/Libvirt#advanced-usage says, that If you need to adjust access controls for a single guest, adjust /etc/apparmor.d/libvirt-, where is the UUID of
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c8 --- Comment #8 from Martin Kalivoda <supercoolemail@seznam.cz> --- (In reply to James Fehlig from comment #6) the guest Which would be totally useless if it would get removed at shutdown. It is also mentioned here https://doc.opensuse.org/documentation/leap/virtualization/html/book.virt/ch... Now, checking the source code and when this removal you mention was introduced: https://github.com/libvirt/libvirt/commit/eba2225bc52624e748cb875e10962bc4c4... and that it is included also in libvirt 4.0.0 which is in Leap 15 it seem that both documentations are wrong.
Since the libvirt apparmor driver is not loaded, the default security model is 'none'. Setting security_default_confined has no effect when the active model is a no-op. This is unfortuates, because comment makes it sound that this "if apparmor is available" applies only to libvirtd, but qemu instances "will be confined by default" when config is set to nonzero (as the condition does not seem to apply to this part as it is different sentence).
I just installed new Tumbleweed with KVM Host and Virtualization Tools pattern. I changed only security_driver, started libvirt and I get the same error. No other changes were made after installation. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c9 --- Comment #9 from James Fehlig <jfehlig@suse.com> --- (In reply to Martin Kalivoda from comment #8)
it seem that both documentations are wrong.
Agreed, but let's stay focused on this bug.
I just installed new Tumbleweed with KVM Host and Virtualization Tools pattern. I changed only security_driver, started libvirt and I get the same error. No other changes were made after installation.
I don't see any problems on my Tumbleweed KVM host, with or without apparmor confinement of VMs. Please answer my questions in #5. We need to figure out why the libvirtd profile was not parsed. -- You are receiving this mail because: You are on the CC list for the bug.
why the libvirtd profile was not parsed. Libvirtd itself is confined and it's supporting utilities (libvirtd//qemu_bridge_helper, virt-aa-helper) are confined too, as stated in my comment #3 (which corrects #2 - it was poorly worded, because it was unclear whether libvirt means both libvirt-* profiled or also libvirtd - I meant just
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c10 --- Comment #10 from Martin Kalivoda <supercoolemail@seznam.cz> --- (In reply to James Fehlig from comment #9) libvirt-*). To make it clear: - libvirtd is confined - apparmor works fine otherwise - failure in clean install can be triggered by explicitly setting security_driver to apparmor, nothing more is necessary.
I don't see any problems on my Tumbleweed KVM host Do you have updated Tumbleweed with libvirt 5 and did you explicitly set security_driver to apparmor in /etc/libvirt/qemu.conf ?
-- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c11 --- Comment #11 from James Fehlig <jfehlig@suse.com> --- (In reply to Martin Kalivoda from comment #10)
Do you have updated Tumbleweed with libvirt 5 and did you explicitly set security_driver to apparmor in /etc/libvirt/qemu.conf ?
Yes. I also have 'security_default_confined = 1'. After starting a VM I see its auto-generated profile has been parsed and loaded (along with other libvirt-based profiles) # aa-status | grep libvirt /usr/sbin/dnsmasq//libvirt_leaseshelper /usr/sbin/libvirtd /usr/sbin/libvirtd//qemu_bridge_helper libvirt-66154842-e926-4f92-92f0-1c1bf61dd1ff /usr/sbin/libvirtd (21446) /usr/bin/qemu-system-x86_64 (21753) libvirt-66154842-e926-4f92-92f0-1c1bf61dd1ff -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c12 --- Comment #12 from Martin Kalivoda <supercoolemail@seznam.cz> --- (In reply to James Fehlig from comment #11)
(In reply to Martin Kalivoda from comment #10) Old profiles in /etc/apparmor.d/libvirt were incorrect (unexpected tokens). I deleted them, but it didn't solve anything... which makes sense, since even clean install does not work for me. Apparently, just setting security_driver to apparmor is not enough to make it work but it used to be enough in near past. Should it still work? I just did anither install of Tumbleweed, this time into VM and I still get the error after setting security_driver to apparmor.
-- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c13 --- Comment #13 from James Fehlig <jfehlig@suse.com> --- Enable debug in /etc/libvirt/libvirtd.conf, restart libvirtd, and see if there is any info indicating why the apparmor driver is not loading. E.g. in /etc/libvirt/libvirtd.conf log_level = 1 log_outputs="1:file:/tmp/libvirtd.log" restart libvird then look for apparmor hints in /tmp/libvirtd.log -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c14 --- Comment #14 from Martin Kalivoda <supercoolemail@seznam.cz> --- Unfortunately it fails between two lines, not meaningful info is there. In the meantime I have examined the source code. My problem comes from here https://github.com/libvirt/libvirt/blob/600462834f4ec1955a9a48a1b6b4a390b9c3... It calls probe on each item of array of externs: security_drivers. Probe of apparmor driver is called AppArmorSecurityManagerProbe: https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997d... It checks if TEMPLATE.qemu and TEMPLATE.lxc exist (both do as it does not print error). Before that it checks for use_apparmor(), since later checks were not even hit (they log errors and if they passed, then I wouldn't get my error), this must be the culprit. Now use_apparmor(): https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997d... First check (virResolveLink) is ok, because it does not print error. Next I dont' use lxc so no prolem here. Next it can probably access APPARMOR_PROFILES_PATH. Then it checks if libvirtd profile exists and is in enforcing mode by calling profile_status(libvirt_daemon, 1). Now profile_status(libvirt_daemon, 1): https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997d... Manually evaluating the code seems to be ok. String building should not fail, access to APPARMOR_PROFILES_PATH is ok, because it does not print error. The, libvirtd line is present and "(enforce)" is present next to it. So no problem here either. So I have started almighty strace and found that it passed readlink test, access test just to fail after https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997d... It must fail after that point, because it didn't log any error. Unfortunately, this would mean that strstr failed for some reason... I'll try to intercept c-library calls to check it out. Also, I don't see any way how it could possibly work for you :/ -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c15 --- Comment #15 from Martin Kalivoda <supercoolemail@seznam.cz> --- Created attachment 797345 --> http://bugzilla.opensuse.org/attachment.cgi?id=797345&action=edit Screenshot of important part of strace log Captured on clean Tumbleweed VM -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c16 --- Comment #16 from Martin Kalivoda <supercoolemail@seznam.cz> --- C library functions were inlined by compiler... ltrace is of no use here. But at least I am sure it fails where I though because I have checked that it reads whole /sys/kernel/security/apparmor/profiles and fails right after that. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c17 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(supercoolemail@se | |znam.cz) | --- Comment #17 from James Fehlig <jfehlig@suse.com> --- (In reply to Martin Kalivoda from comment #14)
So I have started almighty strace and found that it passed readlink test, access test just to fail after https://github.com/libvirt/libvirt/blob/ d56afb8e3997ae19fd7449f773065a2b997dc7c1/src/security/security_apparmor.c#L90 It must fail after that point, because it didn't log any error. Unfortunately, this would mean that strstr failed for some reason... I'll try to intercept c-library calls to check it out.
I suspect this is a regression caused by my commit to change the libvirtd profile to a named profile https://libvirt.org/git/?p=libvirt.git;a=commit;h=a3ab6d42d825499af44b8f19f9...
Also, I don't see any way how it could possibly work for you :/
I had modified /etc/apparmor.d/usr.sbin/libvirtd so it was not overwritten when I updated to libvirt 5.0.0, so in the end I wasn't using the new named profile :-(. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c18 --- Comment #18 from Martin Kalivoda <supercoolemail@seznam.cz> --- (In reply to James Fehlig from comment #17) That must be it! It takes full path to libvirtd here https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997d... and then passes it to profile_status where profile list gets matched against it and matching fails, because profile list contains just profile name. One posiblity should be to remove this https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997d... and replace this https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997d... with rc = profile_status("libvirtd", 1); -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c19 Christian Boltz <suse-beta@cboltz.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |suse-beta@cboltz.de --- Comment #19 from Christian Boltz <suse-beta@cboltz.de> --- To make sure we get this right, can you please check /var/log/audit/audit.log for AppArmor denials (or simply attach that file)? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c20 --- Comment #20 from Martin Kalivoda <supercoolemail@seznam.cz> --- Created attachment 798191 --> http://bugzilla.opensuse.org/attachment.cgi?id=798191&action=edit Audit log from clean VM where libvirtd failed due to this bug (In reply to Christian Boltz from comment #19) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c21 --- Comment #21 from Christian Boltz <suse-beta@cboltz.de> --- (In reply to Martin Kalivoda from comment #20)
Created attachment 798191 [details] Audit log from clean VM where libvirtd failed due to this bug
The log shows that several profiles were loaded, but doesn't show any denials caused by AppArmor. This could mean a) there were no denials ;-) or b) something gets denied by a "deny" rule, which also silences logging. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c22 --- Comment #22 from Martin Kalivoda <supercoolemail@seznam.cz> --- (In reply to Christian Boltz from comment #21) This has nothing to do with apparmor itself and I see no reason to expect denials in audit log. As described in #18: This bug is caused by wrong expectations from libvirt - it expects /usr/sbin/libvirtd in profile lists but profile list only contains "libvirtd" because profile was changed to named profile with name "libvirtd" by commit linked in #17. Code linked in #18 then thinks that libvirtd is not confined and thinks that apparmor is not activated which then results in error from original post. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c23 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CONFIRMED --- Comment #23 from James Fehlig <jfehlig@suse.com> --- (In reply to Martin Kalivoda from comment #22)
(In reply to Christian Boltz from comment #21) This has nothing to do with apparmor itself and I see no reason to expect denials in audit log. As described in #18: This bug is caused by wrong expectations from libvirt - it expects /usr/sbin/libvirtd in profile lists but profile list only contains "libvirtd" because profile was changed to named profile with name "libvirtd" by commit linked in #17. Code linked in #18 then thinks that libvirtd is not confined and thinks that apparmor is not activated which then results in error from original post.
Yep, it's a libvirt problem. Sorry I haven't had time to create a proper fix. Hopefully before the week is out... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1125841 http://bugzilla.opensuse.org/show_bug.cgi?id=1125841#c24 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|CONFIRMED |IN_PROGRESS --- Comment #24 from James Fehlig <jfehlig@suse.com> --- I sent a small patch series upstream to fix this issue and another one I encountered while testing VM confinement with the named libvirtd profile https://www.redhat.com/archives/libvir-list/2019-March/msg00053.html -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com