Sounds like PulseAudio's flat-volume feature. Try to turn off the flat volume by editing /etc/pulse/daemon.conf and set flat-volumes = no