Oh, I just realized that you seem to be using xgifb for the kernel's console mode. Maybe you can try sisfb instead? As for the mode setting when the X server exits, there may be a clash between the drivers. Please try sisfb, as well as not using a kernel frame buffer driver at all (just plain old VGA 80x25 text mode).