Could you try to test XFCE with the smaller memory size as I did? In my case with KDE, mem=1G sufficed to trigger the problem quickly. You can try a slightly smaller value, too.