So now that pipewire sees the ALSA devices the automatic mute and level change on plug/unplug happens again. Which again breaks the speaker because it uses the headphone level. However, I can mute and unmute every control individually, so at least one bug was fixed. I think it was possible to mute the speaker independently of the headphones at some point so I will try if it works with a very old kernel.