Interestingly enough even with swiotlb=force on the kernel command line I get a working vc4 fb allocation (8MB).