This problem was never solved and appears to be a bug in Pulse Audio. I managed to switch to Pipewire and the problem does not exist with Pipewire.