> My guess is that you might have hit a problem with an outdated cache file
> in /var/cache/apparmor/ (the cache is only checked based on the
> timestamps of the involved files in /etc/apparmor.d/).
> However, I'm still a bit surprised because the profiles should have a
> new-enough timestamp (there have been some AppArmor changes unrelated to
> the profiles, but as part of that, the profiles get a newer timestamp when
> the package gets built.)
>
> Another possible part of the puzzle is the precompiled cache in
> /usr/share/apparmor/cache/ - maybe your autogenerated samba profile
> sniplet was "too old" so that the precompiled cache "won"?
> But again, there were no recent changes to the profiles, so in this case
> it {w,sh}ould have been broken for you since quite a while.
>
>
> Hmm, maybe I have an explanation. It sounds like a combination of corner
> case and bad luck, but it's still more likely than the guesses above:
>     Did the breakage start with the reboot after a kernel update?

Yes, it started right after I rebooted as prior to do the update one of my
other TW machines which had been updated to 20221101 was still connected to the
server which was still running 20220908 ( but had just been updated but not rebooted yet)

> (Since you came from 20220908, it's likely that you had kernel 5.19.x
> before, and now have 6.0.x. If in doubt, check /var/log/zypp/history for
> the date of kernel updates.)
> I see that a recent kernel update (probably 6.0.x, installed around
> 2022-10-07) got a new cache hash (which means it has support for new
> AppArmor features), which also means a fresh cache file in
> /var/cache/apparmor/$hash/ will be created. But - only if a profile or
> include file is newer than the precompiled cache. With your timestamp
> from last year, that would not have been the case, and the precompiled
> cache might have been used.

I checked and you are correct.

When I was running 20220908 I was using kernel 5.19.7-1.1 and after updating
to 20221101 I was using kernel 6.0.5-1.1.

> I already wanted to ask for a backup of /var/cache/apparmor/ in the
> broken state, but then checked on my own system (I have samba installed,
> but only start/use it every few months.)
>
> # ls -l /usr/share/apparmor/cache/201d1af9.0/*smbd   \
>     /var/cache/apparmor/*/*smbd
> [...]  4. Nov 18:23 /usr/share/apparmor/cache/201d1af9.0/usr.sbin.smbd
> [...]  7. Okt 21:40 /var/cache/apparmor/3478577f.0/usr.sbin.smbd
>
> Note that there is no /var/cache/apparmor/201d1af9.0/usr.sbin.smbd - my
> autogenerated profile sniplet is too old:
> [...] 13. Apr 2022  /etc/apparmor.d/local/usr.sbin.smbd-shares
> and therefore /usr/share/apparmor/cache/201d1af9.0/usr.sbin.smbd is
> newer and looks "valid enough" :-/

> OK, so we have an explanation.
>
>
> I guess that's one more reason to remind upstream to move "use a
> checksum of the /etc/apparmor.d/ files when validating the cache" more to
> the top of the TODO list.

Here are the results from my system

# ls -l /usr/share/apparmor/cache/201d1af9.0/*smbd /var/cache/apparmor/*/*smbd

-rw-r--r-- 1 root root 45K Oct 10 15:02 /usr/share/apparmor/cache/201d1af9.0/usr.sbin.smbd
-rw------- 1 root root 45K Nov  3 22:41 /var/cache/apparmor/201d1af9.0/usr.sbin.smbd
-rw------- 1 root root 42K Jul 22  2021 /var/cache/apparmor/2cfa59e0.0/usr.sbin.smbd
-rw------- 1 root root 45K Oct 10 15:02 /var/cache/apparmor/3478577f.0/usr.sbin.smbd
-rw------- 1 root root 43K Nov 16  2021 /var/cache/apparmor/b4df858c.0/usr.sbin.smbd


>>>> Unless I overlooked something, the DENIED lines were for two files:
>> I did not give you all the denied lines but that had access denied.
>>
>> Looking at the log I can break the denied lines into 3 groups
>>
>> 1) /var/lib/nscd/netgroup
>> 2) files on smb shares which I have setup
>> 3) lines which are for samba-dcerpcd
>>
>> Here are some examples of the samba-dcerpcd log lines:
>>
>> type=AVC msg=audit(1662039375.496:2612): apparmor="DENIED"
>> operation="file_inherit" profile="samba-dcerpcd"
>> name=2F766F6C2F672F4465736B746F702F53686F7720746F2044656E6973652F
>
> In case you wonder - that cryptic name= can be decoded with aa-decode.
> In this case, it's a directory starting with   /vol/g/   and including a
> space (which is the reason why it was encoded) - so it sums up with "file
> on the shares".

Thanks for the info on aa-decode.

I also found ausearch -i will convert the audit timestamps to human readable.

>>>> If the only new log events you still get are for
>>>> /var/lib/nscd/netgroup, a mail reply is enough. If you get
>>>> more/different log lines, please open a bugreport and attach your
>>>> audit.log.
>>
>> Correct no denied errors since I restart the services.
>
> Does that mean you also don't get denials for /var/lib/nscd/netgroup  ?
> If not, maybe it was related to the strange things you've seen, and we
> don't need to allow it?

No, it still has DENIED for /var/lib/nscd/netgroup.

None for today 11/15/2022 but here are the ones for the last 3 days

type=AVC msg=audit(11/12/22 14:40:33.855:2363) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=5876 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/12/22 15:45:40.592:2368) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=8010 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/12/22 16:02:46.111:2509) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=9657 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/12/22 16:09:05.773:2510) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=11716 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/12/22 16:10:02.687:2511) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=12305 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/12/22 16:17:35.934:2512) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=13866 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/12/22 16:18:21.973:2513) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=14018 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/12/22 16:37:19.127:2531) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=16458 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/12/22 16:40:38.650:2532) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=16626 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/12/22 16:44:11.904:2535) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=16840 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/12/22 16:47:22.615:2536) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=17500 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/13/22 10:51:01.809:2760) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=12575 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/13/22 11:18:16.630:2778) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=14627 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/13/22 14:56:43.959:2844) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=1606 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/14/22 10:09:35.969:3021) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=23281 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root
type=AVC msg=audit(11/14/22 16:39:15.600:3090) : apparmor=DENIED operation=file_receive profile=smbd name=/var/lib/nscd/netgroup pid=14218 comm=smbd requested_mask=r denied_mask=r fsuid=root ouid=root



>
>> So do we have an explanation as to what occurred?   Seems odd that the
>> files compare and that the only change is that the smb.conf
>> got a new timestamp causing apparmor to reload the both files still
>> compared to before the update.
>>
>> So I guess if it happened again, I would just touch smb.conf causing
>> apparmor to reload and then it would work again?
>
> Right.
> And before doing that, please save /etc/apparmor.d/,
> /usr/share/apparmor/cache/ and/var/cache/apparmor/ as a tar.gz so that
> we can easily check for differences and timestamps.

Ok, thanks, I've made a note of that.