I uninstalled Mesa-dri-nouveau and I haven't seen the issue again. Unfortunately I also switched to one GPU simultaneously, so I did 2 changes at once. I suspect Mesa-dri-nouveau was the culprit, but not 100% sure. I will retry with 2 GPUs in a couple of weeks. Feel free to wait or close now (I will re-open if the problem reappears with 2 GPUs).