It seems timeout to do a DAPM powerdown after a stream is closed is 5 seconds by default (see pmdown_time), hence the 5 second timeout I guess. If that is really the issue, not sure why it affects my system though.