Only starting X server on a second virtual console seems temporary work. Switching consoles might lead to a crash and starting an X server on the same virtual console does crash immediately :(((