Just as an update or (hopefully) final comment: Setting the kernel parameter `nvme_core.default_ps_max_latency_us=5500` helped and I haven't seen this issue since.