Hmm ... the package texlive-filesystem should always create the group mktex ... do you have the programs /usr/bin/getent and /usr/sbin/groupadd ... and the pacakge does pre require /usr/sbin/groupadd as well as /usr/bin/getent with Requires(pre) tag as the pre install scriplet does execute both of them in install case