After some more testing the problem appears only on NFS shares (using same username and ID as the nfs server, auto-mounted with autofs).