There is already a fix (mainline and in 6.0.9) which should address this issue: drm/vc4: hdmi: Fix HSM clock too low on Pi4