This should have been improved with the latest work that Qu did on qgroups. His patches seem to have been backported to SLE12-SP3 and newer and SLE15 and newer. Additionally Mark Fasheh is working on further improvement to the generic backref walking logic.