http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c2 --- Comment #2 from Ralf K�lmel <ralf.koelmel@kit.edu> --- the most recent apparmor packages are installed: rpm -qa 'apparmor*' apparmor-abstractions-2.13.6-150300.3.11.2.noarch apparmor-utils-2.13.6-150300.3.11.2.noarch apparmor-parser-lang-2.13.6-150300.3.11.2.noarch apparmor-utils-lang-2.13.6-150300.3.11.2.noarch apparmor-parser-2.13.6-150300.3.11.2.x86_64 apparmor-docs-2.13.6-150300.3.11.2.noarch apparmor-profiles-2.13.6-150300.3.11.2.noarch -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c4 --- Comment #4 from Christian Boltz <suse-beta@cboltz.de> --- Hmm, that's interesting[tm] - in theory it should work with these packages. The log in your initial bugreport looks like you didn't have the updated profiles yet, therefore a few questions: - do you still see the same DENIED lines in /var/log/audit/audit.log ? - can you show the lines for the samba and apparmor packages from /var/log/zypp/history ? (The timestamp is relevant for the next question.) - when the profiles were reloaded during installing the update, does audit.log say operation="profile_replace" info="same as current profile, skipping" or just operation="profile_replace" for the smbd profile? (If it has "same as current profile", there might be something interesting with your cache.) See [1] below regarding timestamps in audit.log. Speaking about the cache - I'd also be interested in the result of ls -l /etc/apparmor.d/usr.sbin.smbd /etc/apparmor.d/local/usr.sbin.smbd* /var/cache/apparmor/*/usr.sbin.smbd to see the timestamps of your profile, included local files, and the cache files. With some luck, the output of above ls already explains the problem - my wild guess is that your /etc/apparmor.d/local/usr.sbin.smbd-shares could be newer than the packaged profile, and therefore the cache looks "new enough". [1] If you want to search for the audit.log entries at the time of installing the update yourself, you might wonder where the timestamp in audit.log hides - it's at the beginning of the line at msg=audit(1644003177 (as unix timestamp) and date -d @1644003177 will translate it to a human-readable date. For the other way round (translating the timestamp from zypper.log to a unix timestamp) you can use date +%s -d '2022-02-04 20:05' If you don't want to search for the relevant audit.log timestamp, just provide the zypp/history sniplet and attach your full audit.log. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c6 --- Comment #6 from Ralf K�lmel <ralf.koelmel@kit.edu> --- Created attachment 855934 --> http://bugzilla.opensuse.org/attachment.cgi?id=855934&action=edit audit log of smb/nmb restart after apparmor cache deletions also after apparmor cache deletions the same apparmor DENIED messages are still present -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c8 --- Comment #8 from Christian Boltz <suse-beta@cboltz.de> --- From the log in the original report:

type=AVC msg=audit(1643828237.305:807): apparmor="DENIED" operation="open" profile="smbd" name="/etc/ssl/openssl.cnf" pid=6144 comm="smbd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

IIRC I've never seen Samba trying to read the openssl.cnf (but I have to admit that I use it only rarely). Do you have a special/unusual samba config that could explain it? @Noel: if you have an idea, feel free to answer as well ;-)

type=AVC msg=audit(1643828237.385:809): apparmor="DENIED" operation="exec" profile="smbd" name="/usr/lib64/samba/samba-bgqd" pid=6148 comm="smbd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0

I assume that was fixed by deleting the cache? Also, if you see other DENIED events, please tell me. Unfortunately I'm still not sure why you needed to delete the cache. The "same as current profile, skipping" log entries from comment 5 happened at the same time as the updated apparmor-profiles package was installed. This means AppArmor thought the cache was new enough. While this happens on the kernel side, it could also be caused by apparmor_parser - if it thinks the cache is up to date, it just passes the cache to the kernel. (The cache is checked based on the timestamp of the profile and all included files, unfortunately not based on the content of those files) My guess was that you might have a /etc/apparmor.d/local/usr.sbin.smbd-shares (generated from your smb.conf) that is newer than the new packaged smbd profile - but your usr.sbin.smbd-shares is a year old, so my guess doesn't fit your case. Would have been too easy ;-) and I'm somewhat afraid that in your case it might stay a mystery what exactly happened. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c9 --- Comment #9 from Noel Power <nopower@suse.com> --- (In reply to Christian Boltz from comment #8)

From the log in the original report:

...

type=AVC msg=audit(1643828237.305:807): apparmor="DENIED" operation="open" profile="smbd" name="/etc/ssl/openssl.cnf" pid=6144 comm="smbd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

IIRC I've never seen Samba trying to read the openssl.cnf (but I have to admit that I use it only rarely). Do you have a special/unusual samba config that could explain it?

@Noel: if you have an idea, feel free to answer as well ;-) no idea, sorry, I'm guessing this must be pulled in by some external library used by samba. I've cc'ed samba-maintainers so maybe someone else here might have an idea

...

type=AVC msg=audit(1643828237.385:809): apparmor="DENIED" operation="exec" profile="smbd" name="/usr/lib64/samba/samba-bgqd" pid=6148 comm="smbd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0

I assume that was fixed by deleting the cache?

Also, if you see other DENIED events, please tell me.

Unfortunately I'm still not sure why you needed to delete the cache.

The "same as current profile, skipping" log entries from comment 5 happened at the same time as the updated apparmor-profiles package was installed. This means AppArmor thought the cache was new enough. While this happens on the kernel side, it could also be caused by apparmor_parser - if it thinks the cache is up to date, it just passes the cache to the kernel. (The cache is checked based on the timestamp of the profile and all included files, unfortunately not based on the content of those files)

My guess was that you might have a /etc/apparmor.d/local/usr.sbin.smbd-shares (generated from your smb.conf) that is newer than the new packaged smbd profile - but your usr.sbin.smbd-shares is a year old, so my guess doesn't fit your case. Would have been too easy ;-) and I'm somewhat afraid that in your case it might stay a mystery what exactly happened.

I have experienced cache related problems a couple of times recently, however every time I try to pin it down and reproduce it I have failed :/ -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c11 --- Comment #11 from Christian Boltz <suse-beta@cboltz.de> --- (In reply to Ralf K�lmel from comment #10)

I'm still having "denied" messages during smb/nmb startup(s. attached file) , but they don't break the smb service start.

That log basically shows 3 [groups of] denials: - capability net_admin -> bug 1196922 aka bug 1196850 comment #3 - /proc/*/fd/ -> https://gitlab.com/apparmor/apparmor/-/merge_requests/860 - smbd and samba-bgqd reading /etc/ssl/openssl.cnf -> this bugzilla comment Allowing to read the openssl config is quite harmless. Also, samba.spec contains BuildRequires: libopenssl-devel so reading openssl.cnf isn't too surprising. Long story short: I just submitted https://gitlab.com/apparmor/apparmor/-/merge_requests/862

The question is if automatic cleaning of the apparmor cache (through configuration in the rpm spec) after an update of the apparmor packages would be a pragmatic way to workaround the problem.

That workaround would be rm /var/cache/apparmor/*/usr.sbin.smbd but I'm not sure if I like it in a %post script ;-) (would need to be done before restarting samba)

For now i'm afraid to do automatic updates (of apparmor packages) on affected systems because it could break the samba daemon restart.

Understandable. If it helps: - you can run the above workaround before updating - this is the first time I've seen such an issue, therefore I'd say the risk for similar problems in future updates is quite low. (Yes, famous last words ;-)

But besides a workaround the best would be to find the problem.

Agreed. (In reply to Noel Power from comment #9)

I have experienced cache related problems a couple of times recently, however every time I try to pin it down and reproduce it I have failed :/

Timestamps of the profiles and indirectly also your samba config (via the autogenerated profile sniplet) are relevant, which makes reproducing harder. If that happens again, please save the following files and directories in a tarball (timestamps are most important, but in worst case it's also possible to investigate the content of the cache files): - /etc/samba/smb.conf (or at least its timestamp) - /etc/apparmor.d/ - /var/cache/apparmor/ - /usr/share/apparmor/cache/ - /var/log/audit/audit.log -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c13 Noel Power <nopower@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |suse-beta@cboltz.de Flags| |needinfo?(suse-beta@cboltz. | |de) --- Comment #13 from Noel Power <nopower@suse.com> --- (In reply to Noel Power from comment #12)

(In reply to Christian Boltz from comment #11) [...]

Thanks for pointing out the important information to save, I will keep an eye out, hopefully I will experience this again soon so I remember this bug and/or the instructions :-)

Hi Christian, I reproduced this, better still I think I can keep on reproducing it thanks to a vm (snapshotted before/after) this is Leap 15.4 (not probably fully updated) but I attempted to update with https://build.opensuse.org/package/show/home:npower:branches:security:apparm... (adjusts apparmor with /proc/{pipd}/fd rule) What is happening is the cached file /etc/apparmor.d/cache.d/*/samba-bgqd doesn't appear to be updated correctly with the install on system localhost:~ # ls -lrt /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd -rw------- 1 root root 37305 Mar 23 20:43 /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd localhost:~ # ls -lrt /usr/share/apparmor/cache/d29c4283.0/samba-bgqd -rw-r--r-- 1 root root 37345 Mar 23 15:50 /usr/share/apparmor/cache/d29c4283.0/samba-bgqd but it *did* update it (see date) if I stop apparmor, remove /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd and restart the cache file size matches /usr/share/apparmor/cache/d29c4283.0/samba-bgqd and I no longer get the DENIED for /proc/{pipd}/fd I will attach a tarball with the data you requested. Let me know if you need any other info and/or debugging I can do to help -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c15 Christian Boltz <suse-beta@cboltz.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(suse-beta@cboltz. | |de) | --- Comment #15 from Christian Boltz <suse-beta@cboltz.de> --- (In reply to Noel Power from comment #13)

Hi Christian, I reproduced this, better still I think I can keep on reproducing it thanks to a vm (snapshotted before/after)

Sounds promising :-) and even if I don't yet know exactly what's going on, that should help with debugging.

this is Leap 15.4 (not probably fully updated) but I attempted to update with https://build.opensuse.org/package/show/home:npower:branches:security: apparmor/apparmor (adjusts apparmor with /proc/{pipd}/fd rule)

Looks good, feel free to send a SR ;-)

What is happening is the cached file /etc/apparmor.d/cache.d/*/samba-bgqd doesn't appear to be updated correctly with the install on system

Just for the records: on openSUSE, /etc/apparmor.d/cache.d/ is a symlink to /var/cache/apparmor/ (IIRC some other distros have the cache directly in /etc/apparmor.d/cache.d/) I'll paste the relevant files/timestamps from the tarball (ignoring abstractions for now) in a slightly more readable format: Mar 23 16:51 ./etc/apparmor.d/samba-bgqd Mar 23 16:51 ./etc/apparmor.d/local/samba-bgqd Mar 23 16:50 ./usr/share/apparmor/cache/d29c4283.0/samba-bgqd Mar 23 21:43 ./var/cache/apparmor/d29c4283.0/samba-bgqd This means: - the packaged cache in /usr/ has a too-old timestamp (that's a packaging bug, and I have a solution mostly ready - just waiting for an upstream answer for a detail question) - the locally generated cache in /var/ looks new enough (clearly newer than the profile in /etc/)

but it *did* update it (see date)

Indeed, and that's the strange part.

if I stop apparmor, remove /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd and restart the cache file size matches /usr/share/apparmor/cache/d29c4283.0/samba-bgqd and I no longer get the DENIED for /proc/{pipd}/fd

Just deleting the cache file in /var/ + rcapparmor reload should be enough - no need to stop apparmor.

I will attach a tarball with the data you requested. Let me know if you need any other info and/or debugging I can do to help

Just to be sure - does your tarball contain the "broken" or the "working" cache? I'd also be interested in an audit.log that includes reloading the samba-bgqd profile multiple times and in multiple cases? Please first read the "Crazy idea" at the end, then run sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log rcapparmor reload sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log echo "zypper up" >> /var/log/audit/audit.log zypper up # or whatever you need to install your updated AppArmor packages sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log rcapparmor reload sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log echo "samba-bgqd cache deleted" >> /var/log/audit/audit.log rm -v /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log rcapparmor reload sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log With this log spamming ;-) audit.log should show where things break. /var/log/zypp/history (the section with installing your updated AppArmor packages is enough) might also be helpful. Crazy idea, since you already have your own branch packages: can you please add sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log sleep 5 in %post of apparmor-parser, apparmor-profiles and apparmor-abstractions before doing the above test? That sha256sum+sleep should be the last thing %post does. (wild guess: both -profiles and -abstractions reload the profiles, and I wonder if that could be part of the problem. The first one triggers cache generation, and second one changes a file in /etc - with same timestamp as all packaged profiles, so that the cache doesn't get regenerated.) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c17 --- Comment #17 from OBSbugzilla Bot <bwiedemann+obsbugzillabot@suse.com> --- This is an autogenerated message for OBS integration: This bug (1195463) was mentioned in https://build.opensuse.org/request/show/964948 Factory / apparmor -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c19 --- Comment #19 from Noel Power <nopower@suse.com> --- Created attachment 857476 --> http://bugzilla.opensuse.org/attachment.cgi?id=857476&action=edit tar of apparmor log (and zypper history) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c21 --- Comment #21 from Noel Power <nopower@suse.com> --- (In reply to Christian Boltz from comment #20)

Hmm, in this case only deleting the cache file helped :-/ I know it's hard to find out in hindsight, but do you remember if the cache file you deleted had a newer timestamp than the profile in the apparmor-profiles rpm installed during zypper up?

no but I should be able to reproduce this and tell you

Anyway, I think there are two things I need to fix:

1. The timestamp of the precompiled cache (/usr/share/apparmor/cache/) is older than the profiles. cp -r instead of cp -a for copying the cache in %install will fix that, and should prevent generation of cache files in /var/cache/apparmor (you might need to delete the content of /var/cache/apparmor to see this effect - nevertheless I'm not sure if I should do that in %post. Note that this is just about getting rid of some files in /var/cache/apparmor, but together with 2 the cache file shouldn't hurt.)

2. Reloading the profiles (which also means updating the cache) is done in %post of apparmor-profiles _and_ apparmor-abstractions. Unfortunately the first reload happens when only one of the packages got updated, therefore the cache is based on a half-updated state. When the second package gets updated, the cache is new enough (newer than the packaged profiles) and won't get updated again. The change for 1 makes it less likely that this happens, but to be sure, I'll move reloading the profiles from %post to %posttrans.

I just submitted a package with these fixes to home:cboltz/apparmor. Can you please test if you still can reproduce the issue with these packages? (Please do NOT delete any cache files manually before installing the packages to get a real-world result. You should get updated cache files in /var/cache/apparmor/, probably with the same content/checksum as in /usr/share/apparmor/cache/.)

will try after (re)trying the debug package to get your answer above -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c23 --- Comment #23 from Noel Power <nopower@suse.com> --- (In reply to Noel Power from comment #21) u please test if you still can reproduce the issue with these packages?

...

(Please do NOT delete any cache files manually before installing the packages to get a real-world result. You should get updated cache files in /var/cache/apparmor/, probably with the same content/checksum as in /usr/share/apparmor/cache/.)

will try after (re)trying the debug package to get your answer above

#before install of your latest localhost:~ # ls -lrt /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd -rw------- 1 root root 37305 Mar 23 16:43 /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd localhost:~ # ls -lrt /usr/share/apparmor/cache/d29c4283.0/samba-bgqd -rw-r--r-- 1 root root 37305 Feb 16 21:25 /usr/share/apparmor/cache/d29c4283.0/samba-bgqd #after install localhost:~ # ls -lrt /usr/share/apparmor/cache/d29c4283.0/samba-bgqd -rw-r--r-- 1 root root 38145 Mar 30 10:12 /usr/share/apparmor/cache/d29c4283.0/samba-bgqd localhost:~ # ls -lrt /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd -rw------- 1 root root 38145 Mar 30 11:30 /etc/apparmor.d/cache.d/d29c4283.0 so.... looking good!!! -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1195463 http://bugzilla.opensuse.org/show_bug.cgi?id=1195463#c32 Christian Boltz <suse-beta@cboltz.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |IN_PROGRESS --- Comment #32 from Christian Boltz <suse-beta@cboltz.de> --- AFAIK this should be fixed in the meantime, but given the quite interesting[tm] problem - is there anything left, or can I close the bug as fixed? -- You are receiving this mail because: You are on the CC list for the bug.