Right, basically it's a job of session manager. The xfce4 session manages seats via systemd logind, and the processes should have been terminated along with it. So something wrong with seat management in xfce4-session side, I suppose. The systemd KillUserProcess=yes is no solution. If XFCE relies on it, it's a clear design mistake.