[Bug 1120472] New: [Build 20181231] openQA test fails in aa_logprof
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472 Bug ID: 1120472 Summary: [Build 20181231] openQA test fails in aa_logprof Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: Other URL: https://openqa.opensuse.org/tests/822549/modules/aa_lo gprof/steps/24 OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: AppArmor Assignee: suse-beta@cboltz.de Reporter: guillaume.gardet@arm.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- ## Observation openQA test in scenario opensuse-Tumbleweed-DVD-aarch64-apparmor@aarch64 fails in [aa_logprof](https://openqa.opensuse.org/tests/822549/modules/aa_logprof/steps/24) ## Reproducible Fails since (at least) Build [20181231](https://openqa.opensuse.org/tests/822362) ## Expected result Last good: [20181224](https://openqa.opensuse.org/tests/821330) (or more recent) ## Further details Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=aarch64&test=apparmor&version=Tumbleweed&distri=opensuse&machine=aarch64&flavor=DVD) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c1
Christian Boltz
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c2
--- Comment #2 from Guillaume GARDET
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c3
Guillaume GARDET
That's interesting[tm], and it looks like openQA found a scenario that I thought would never happen. (Actually I wasn't even aware that we have AppArmor tests in openQA ;-) - thanks to whoever added them!)
I just tried to reproduce the issue locally by doing what openQA does, but (un)fortunately aa-logprof "just works" as expected.
Since it is only in latest openQA snapshot (20181231), it is not in released Tumbleweed.
I have two questions:
Is there a way to get the debug log /tmp/apparmor-bugreport-*.txt from openQA? (note that the filename is mktemp-generated) The debug log contains the content of variables etc. and is hopefully helpful to find out what happens.
Added in attachment.
Also, where can I find the source code of the AppArmor tests done by openQA? Maybe there's a detail I missed in the screenshot.
You can get by clicking aa_logprof (test title) in openQA: https://openqa.opensuse.org/tests/822549#step/aa_logprof/24 BTW, in latest Tumbleweed snapshot, it seems '/usr/sbin/nscd' has been replaced by just 'nscd'. Not sure if it could be the cause? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c4
--- Comment #4 from Christian Boltz
(In reply to Christian Boltz from comment #1)
I just tried to reproduce the issue locally by doing what openQA does, but (un)fortunately aa-logprof "just works" as expected.
Since it is only in latest openQA snapshot (20181231), it is not in released Tumbleweed.
That's why I'm using the apparmor packages from home:cboltz (which I submitted to security:apparmor and then to Tumbleweed) ;-) (nevertheless: thanks for mentioning it, it would have been an easy explanation ;-) Also, it turned out that I need to run more of the AppArmor tests (not only the aa-logprof test) to reproduce the bug. See below for the full story.
Is there a way to get the debug log /tmp/apparmor-bugreport-*.txt from openQA? (note that the filename is mktemp-generated) The debug log contains the content of variables etc. and is hopefully helpful to find out what happens.
Added in attachment.
Thanks! The most strange, interesting and useful line is profile = '/usr/sbin/nscd'
Also, where can I find the source code of the AppArmor tests done by openQA? Maybe there's a detail I missed in the screenshot.
You can get by clicking aa_logprof (test title) in openQA: https://openqa.opensuse.org/tests/822549#step/aa_logprof/24
Good to know, thanks!
BTW, in latest Tumbleweed snapshot, it seems '/usr/sbin/nscd' has been replaced by just 'nscd'. Not sure if it could be the cause?
It looks like this change combined with the other tests you do trigger the issue. Let me explain: Upstream decided to switch to "named profiles" for various reasons, so the nscd profile is now named just "nscd" (with /usr/{bin,sbin}/nscd as attachment) instead of having /usr/bin/nscd for both name and attachment. However, when creating a new profile with aa-genprof or aa-autodep, the path-based way ("/usr/bin/nscd") gets used. This means after your aa-genprof and aa-autodep tests, two profiles are loaded: "nscd" and "/usr/sbin/nscd". (I've seen you delete the profile file and run "rcapparmor restart", but that doesn't unload the "/usr/sbin/nscd" profile - see the Leap 15 release notes for an explanation.) The path-based /usr/sbin/nscd profile is more specific (no alternation in the path) and therefore gets used when starting nscd. That explains the "profile = '/usr/sbin/nscd'" line from the debug log. However, the profile in /tmp/apparmor.d/ is named "nscd", and a "/usr/sbin/nscd" profile doesn't exist there. That's exactly what aa-logprof tells us in the error (ok, the error message could be more specific, but it's clear when looking at the code). To sum it up - the conditions to trigger this error are: - nscd is running under the "wrong" profile name ("/usr/sbin/nscd") and triggered audit.log entries with this profile name - a file "usr.sbin.nscd" file exists (that's the default filename for the /usr/sbin/nscd profile) - in that file, the profile name is _not_ "/usr/sbin/nscd" I hope the openQA test authors play in the lottery, because you hit this quite unlikely corner case ;-) Now we know what's happening, and I hope you enjoyed reading the full story ;-) IMHO there are two things that should be fixed: a) in AppArmor: I'll either "downgrade" the error to a warning saying Ignoring log event for non-existing profile $name, even if the profile file exists (different profile name?) or simply silently ignore events for non-existing profiles since that is what happens for all non-existing profiles not matching this corner case. b) in the openQA tests: unload the profile before you delete the profile file to ensure you have a clean test setup: apparmor_parser -R /tmp/apparmor.d/usr.sbin.nscd rm /tmp/apparmor.d/usr.sbin.nscd cp -a /etc/apparmor.d/ /tmp/apparmor.d/ apparmor_parser -r /tmp/apparmor.d/ # reload profiles nscd will run unconfined after that, but you are stopping it anyway. (And sadly, openQA will no longer cover that corner case it accidently covered ;-) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
Guillaume GARDET
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
lili zhao
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c6
Wnereiz Z
... To sum it up - the conditions to trigger this error are: - nscd is running under the "wrong" profile name ("/usr/sbin/nscd") and triggered audit.log entries with this profile name - a file "usr.sbin.nscd" file exists (that's the default filename for the /usr/sbin/nscd profile) - in that file, the profile name is _not_ "/usr/sbin/nscd"
The profile in usr.sbin.nscd got a profile name "nscd" (which does not exist before), that maybe the reason why we saw it is on "nscd" in the status. You can check the example /etc/apparmor.d/sbin.klogd I'm going to see how to modify the test script to better match what we want to test. (In reply to Christian Boltz from comment #1)
...
I just tried to reproduce the issue locally by doing what openQA does, but (un)fortunately aa-logprof "just works" as expected.
The current Tumbleweed version has not been rolled to what openQA was running, I guess. You probably need a factory build for testing. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c9
--- Comment #9 from Christian Boltz
(In reply to Christian Boltz from comment #4)
... To sum it up - the conditions to trigger this error are: - nscd is running under the "wrong" profile name ("/usr/sbin/nscd") and triggered audit.log entries with this profile name - a file "usr.sbin.nscd" file exists (that's the default filename for the /usr/sbin/nscd profile) - in that file, the profile name is _not_ "/usr/sbin/nscd"
The profile in usr.sbin.nscd got a profile name "nscd" (which does not exist before), that maybe the reason why we saw it is on "nscd" in the status.
Right, that's exactly the reason. BTW: When openQA deletes the profile and lets aa-autodep or aa-genprof generate a new one, it will be named "/usr/sbin/nscd".
I'm going to see how to modify the test script to better match what we want to test.
IMHO you need to unload the nscd profile from the kernel before deleting it, see comment #4 for details. Also, when testing with the shipped profile, you'll have to expect "nscd" as profile name.
(In reply to Christian Boltz from comment #1)
...
I just tried to reproduce the issue locally by doing what openQA does, but (un)fortunately aa-logprof "just works" as expected.
The current Tumbleweed version has not been rolled to what openQA was running, I guess. You probably need a factory build for testing.
No worries, I tested with a new-enough package ;-) and comment #4 (starting at "Let me explain" explains why I initially wasn't able to reproduce the problem. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c12
Christian Boltz
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c13
Guillaume GARDET
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c14
Christian Boltz
...
Now for the aa_logprof test failure:
...
(I have no idea what $self->aa_tmp_prof_clean does (github link welcome) - but it does for sure not unload the "/usr/sbin/nscd" profile.) aa_tmp_prof_clean does nothing but remove /tmp/apparmor.d but in pre_run_hook, which run before every single cases, apparmor and auditd will be restarted. See
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c15
--- Comment #15 from Wnereiz Z
To fix this, run the following at the end of the aa_autodep and aa_genprof tests: echo '/usr/sbin/nscd {}' | apparmor_parser -R to unload the "/usr/sbin/nscd" profile. Expected result: no output, $? == 0
Does this help to make the tests green? ;-)
Thanks for the tips. I wasn't considered the loaded profile will implicate the profile generated in another profile directory by aa-genprof. I'm going to check. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c16
--- Comment #16 from lili zhao
IMHO there are two things that should be fixed:
a) in AppArmor: I'll either "downgrade" the error to a warning saying Ignoring log event for non-existing profile $name, even if the profile file exists (different profile name?) or simply silently ignore events for non-existing profiles since that is what happens for all non-existing profiles not matching this corner case.
b) in the openQA tests: unload the profile before you delete the profile file to ensure you have a clean test setup:
apparmor_parser -R /tmp/apparmor.d/usr.sbin.nscd rm /tmp/apparmor.d/usr.sbin.nscd cp -a /etc/apparmor.d/ /tmp/apparmor.d/ apparmor_parser -r /tmp/apparmor.d/ # reload profiles
nscd will run unconfined after that, but you are stopping it anyway. (And sadly, openQA will no longer cover that corner case it accidently covered ;-)
Thank you so much for the fixing suggestions for openQA tests, we have opened 2 poo to enhance our test cases. FYI: [sle][security][sle15sp1] apparmor aa_autodep & aa_genprof tests need doing cleanup (https://progress.opensuse.org/issues/45980) [sle][security][sle15sp1] apparmor aa_enforce test needs to be updated to match new behavior in Tumblewwed (https://progress.opensuse.org/issues/45635#change-178487) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c17
--- Comment #17 from Christian Boltz
To fix this, run the following at the end of the aa_autodep and aa_genprof tests: echo '/usr/sbin/nscd {}' | apparmor_parser -R to unload the "/usr/sbin/nscd" profile. Expected result: no output, $? == 0
Thanks for the tips. I wasn't considered the loaded profile will implicate the profile generated in another profile directory by aa-genprof. I'm going to check.
Quick explanation: Not unloading the profile (when deleting it on disk) didn't hurt as long as the shipped profile was named "/usr/sbin/nscd" - that's exactly the name aa-genprof and aa-autodep use. Therefore the deleted-on-disk profile was replaced with the new profile, but the end result "/usr/sbin/nscd profile loaded" was the same. With the named profiles ("nscd") we now have a different profile name, which made the missing profile unload visible. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c19
--- Comment #19 from Christian Boltz
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c20
--- Comment #20 from Christian Boltz
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472
http://bugzilla.opensuse.org/show_bug.cgi?id=1120472#c21
Christian Boltz
participants (1)
-
bugzilla_noreply@novell.com