I just removed from /usr/lib/virtualbox the files VBoxKeyboard.so VBoxOGLrenderspu.so VBoxSDL.so VBoxSharedClipboard.so VBoxSharedCrOpenGL.so and I can run a headless machine on a system without X11 libraries. These are the files that contain references to X11 libraries.