http://bugzilla.opensuse.org/show_bug.cgi?id=1198972 http://bugzilla.opensuse.org/show_bug.cgi?id=1198972#c2 --- Comment #2 from Jon Brightwell <jon.brightwell@zpg.co.uk> --- Left it running over night and it is down to 23KiB/s... At subvol snapshot 6.64GiB {----} 21:42:48 [23.3KiB/s] [89.1KiB/s] 29.16% /proc/kcore 0x7fffc0a45553 k [k] read_extent_buffer 14.69% /proc/kcore 0x7fffc0a3a3ea k [k] trace_event_type_funcs_svcrdma_encode_read+0x7fffff3a5ebe 13.01% /proc/kcore 0x7fffc0a45e3e k [k] map_private_extent_buffer 12.58% /proc/kcore 0x7fffc0a8a6eb k [k] nf_tables_fill_gen_info.isra.55 9.31% /proc/kcore 0x7fffc0a3a16b k [k] print_fmt_xprtrdma_rxprt+0x7fffff3a5ebf 3.97% /proc/kcore 0x7fff94f0c6ed k [k] __radix_tree_lookup 2.04% /proc/kcore 0x7fff94841159 k [k] mark_page_accessed 1.41% /proc/kcore 0x7fffc0a3a08d k [k] print_fmt_xprtrdma_rdch_event+0x7fffff3a5f21 1.38% /proc/kcore 0x7fffc0a131ee k [k] rpcrdma_ep_disconnect 1.36% /proc/kcore 0x7fffc09fb9fd k [k] generic_bin_search.constprop.41 1.16% /proc/kcore 0x7fff9467a567 k [k] read_hpet 1.09% /proc/kcore 0x7fffc09fb8ed k [k] btrfs_comp_cpu_keys 1.01% /proc/kcore 0x7fffc0a44758 k [k] find_extent_buffer 0.83% /proc/kcore 0x7fffc0a8a64a k [k] nf_tables_getobj 0.79% /proc/kcore 0x7fffc0a3a3d1 k [k] trace_event_type_funcs_svcrdma_error_event+0x7fffff3a5ec5 0.77% /proc/kcore 0x7fffc0a3a2e3 k [k] trace_event_type_funcs_svcrdma_wc_receive+0x7fffff3a5eb7 0.72% /proc/kcore 0x7fffc0a3a2f4 k [k] trace_event_type_funcs_svcrdma_post_recv+0x7fffff3a5ea8 0.70% /proc/kcore 0x7fffc0a3a377 k [k] trace_event_type_funcs_svcrdma_dma_map_rwctx+0x7fffff3a5eab 0.65% /proc/kcore 0x7fffc0a3a11e k [k] print_fmt_xprtrdma_connect_class+0x7fffff3a5ef2 Not much has changed. Would a full flamegraph/call graph help? The filesystem setup is 4x NVME in BTRFS RAID 10 root@dbshard1a.prod:~ # btrfs filesystem usage /sqldata Overall: Device size: 13.97TiB Device allocated: 4.04TiB Device unallocated: 9.93TiB Device missing: 0.00B Used: 3.42TiB Free (estimated): 5.20TiB (min: 5.20TiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Data,RAID10: Size:1.79TiB, Used:1.56TiB /dev/nvme0n1 917.00GiB /dev/nvme1n1 917.00GiB /dev/nvme2n1 917.00GiB /dev/nvme3n1 917.00GiB Metadata,RAID10: Size:237.00GiB, Used:158.95GiB /dev/nvme0n1 118.50GiB /dev/nvme1n1 118.50GiB /dev/nvme2n1 118.50GiB /dev/nvme3n1 118.50GiB System,RAID10: Size:16.00MiB, Used:272.00KiB /dev/nvme0n1 8.00MiB /dev/nvme1n1 8.00MiB /dev/nvme2n1 8.00MiB /dev/nvme3n1 8.00MiB Unallocated: /dev/nvme0n1 2.48TiB /dev/nvme1n1 2.48TiB /dev/nvme2n1 2.48TiB /dev/nvme3n1 2.48TiB -- You are receiving this mail because: You are on the CC list for the bug.