It can't be pam_xauth.so as when it is invoked (session phase), the process has already switched to the new user.