Comment # 49 on bug 1004453 from Max Staudt

This seems to stem from the new IO_STRICT_DEVMEM option introduced in kernel
v4.5 - it disallows userspace /dev/mem access to regions used by the kernel.
This explains why you can start X with xf86-video-mga when booting in VGA 80x25
text mode, but not when booting with vesafb.

Does booting with iomem=relaxed fix your X?

If yes, please add iomem=relaxed to your boot parameters when using a v4.5
kernel or newer. This is basically the other thing to do after setting the X
server setuid root in order to make old user mode setting drivers (such as
xf86-video-mga) work.

This change in the kernel has been introduced for security reasons, just like
the DRI1 drivers are no longer included by default and the X server runs in
user mode. Since there is nobody using the DRI1 driver for 3D, and thus no need
to build the mga.ko kernel module, I think that setting this kernel option is
the most comfortable workaround for you.