Well, I still have usbcore.autosuspend=-1 in the kernel cmd line. I will try without it this weekend.