Hm, falling back to gvim doubles startup time, though it's still way to small to be noticed. Looking at comment #3 I still think X is not pulled into the dependencies when using --with-x=no. If that holds, that would be my prefered way.