IMO the only possible fix is to split out the intree cpan modules into packages on their own, that can be replaced by binary packages next to it. But mls is not a fan of it.