http://bugzilla.opensuse.org/show_bug.cgi?id=1191929 Bug ID: 1191929 Summary: NVMe SSD slow since openSUSE-LEAP-15.3 (especially with cryptsetup / dm-crypt device) Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.3 Hardware: x86-64 OS: openSUSE Leap 15.3 Status: NEW Severity: Normal Priority: P5 - None Component: Kernel Assignee: kernel-bugs@opensuse.org Reporter: duge@pre-sense.de QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- I got a system with a NVMe SSD, whose performance with dm-crypt dropped from > 200 MB/s to below 2 MB/s from openSUSE-15.2 to openSUSE-15.3. Test procedure: dd oflag=direct bs="$((64 * 1024 * 1024))" count="$((16 * 1024 / 64))" if=dev/zero of=/dev/nvme0n1 dd oflag=sync bs="$((64 * 1024 * 1024))" count="$((16 * 1024 / 64))" if=dev/zero of=/dev/nvme0n1 cryptsetup --cipher "aes-cbc-essiv:sha256" create crypt_test /dev/nvme0n1 dd oflag=direct bs="$((64 * 1024 * 1024))" count="$((16 * 1024 / 64))" if=dev/zero of=/dev/mapper/crypt_test dd oflag=sync bs="$((64 * 1024 * 1024))" count="$((16 * 1024 / 64))" if=dev/zero of=/dev/mapper/crypt_test For oflag=direct openSUSE 15.2 and 15.3 both write with over 800 MB/s to /dev/nvme0n1 and over 300 MB/s to /dev/mapper/crypt_test. openSUSE-15.2 will write with over 600 MB/s to /dev/nvme0n1 and over 200 MB/s to /dev/mapper/crypt_test openSUSE-15.3 will write with 100 MB/s to /dev/nvme0n1 and less than 2 MB/s to /dev/mapper/crypt_test. Finally an "mkfs.ext4 /dev/mapper/crypt_test" takes more than 20 minutes and "iotop" shows even less I/O than 1 MB/s! Tumbleweed-x86_64-Snapshot20211019 (Linux-5.14.11) performs like openSUSE-15.2 or even better. localhost:~ # grep PRETTY /etc/os-release PRETTY_NAME="openSUSE Leap 15.3" localhost:~ # lsblk /dev/nvme0n1 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 238,5G 0 disk localhost:~ # lspci -v -d 1344:5410 04:00.0 Non-Volatile memory controller: Micron Technology Inc Device 5410 (rev 01) (prog-if 02 [NVM Express]) Subsystem: Micron Technology Inc Device 0100 Flags: bus master, fast devsel, latency 0, IRQ 16, NUMA node 0 Memory at a2100000 (64-bit, non-prefetchable) [size=16K] Capabilities: [80] Express Endpoint, MSI 00 Capabilities: [d0] MSI-X: Enable+ Count=32 Masked- Capabilities: [e0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [f8] Power Management version 3 Capabilities: [100] Vendor Specific Information: ID=1556 Rev=1 Len=008 Capabilities: [108] Latency Tolerance Reporting Capabilities: [110] L1 PM Substates Capabilities: [200] Advanced Error Reporting Capabilities: [300] #19 Kernel driver in use: nvme Kernel modules: nvme localhost:~ # smartctl -a /dev/nvme0n1 [...] === START OF INFORMATION SECTION === Model Number: Micron_2200_MTFDHBA256TCK [...] Firmware Version: P1MU003 PCI Vendor/Subsystem ID: 0x1344 IEEE OUI Identifier: 0x00a075 Controller ID: 0 Number of Namespaces: 1 Namespace 1 Size/Capacity: 256.060.514.304 [256 GB] Namespace 1 Formatted LBA Size: 512 Namespace 1 IEEE EUI-64: 00a075 012cd592d5 [...] -- You are receiving this mail because: You are the assignee for the bug.