Comment # 2
on bug 1198972
from Jon Brightwell
Left it running over night and it is down to 23KiB/s...
At subvol snapshot
6.64GiB {----} 21:42:48 [23.3KiB/s] [89.1KiB/s]
29.16% /proc/kcore
0x7fffc0a45553 k [k] read_extent_buffer
14.69% /proc/kcore
0x7fffc0a3a3ea k [k]
trace_event_type_funcs_svcrdma_encode_read+0x7fffff3a5ebe
13.01% /proc/kcore
0x7fffc0a45e3e k [k] map_private_extent_buffer
12.58% /proc/kcore
0x7fffc0a8a6eb k [k] nf_tables_fill_gen_info.isra.55
9.31% /proc/kcore
0x7fffc0a3a16b k [k] print_fmt_xprtrdma_rxprt+0x7fffff3a5ebf
3.97% /proc/kcore
0x7fff94f0c6ed k [k] __radix_tree_lookup
2.04% /proc/kcore
0x7fff94841159 k [k] mark_page_accessed
1.41% /proc/kcore
0x7fffc0a3a08d k [k] print_fmt_xprtrdma_rdch_event+0x7fffff3a5f21
1.38% /proc/kcore
0x7fffc0a131ee k [k] rpcrdma_ep_disconnect
1.36% /proc/kcore
0x7fffc09fb9fd k [k] generic_bin_search.constprop.41
1.16% /proc/kcore
0x7fff9467a567 k [k] read_hpet
1.09% /proc/kcore
0x7fffc09fb8ed k [k] btrfs_comp_cpu_keys
1.01% /proc/kcore
0x7fffc0a44758 k [k] find_extent_buffer
0.83% /proc/kcore
0x7fffc0a8a64a k [k] nf_tables_getobj
0.79% /proc/kcore
0x7fffc0a3a3d1 k [k]
trace_event_type_funcs_svcrdma_error_event+0x7fffff3a5ec5
0.77% /proc/kcore
0x7fffc0a3a2e3 k [k]
trace_event_type_funcs_svcrdma_wc_receive+0x7fffff3a5eb7
0.72% /proc/kcore
0x7fffc0a3a2f4 k [k]
trace_event_type_funcs_svcrdma_post_recv+0x7fffff3a5ea8
0.70% /proc/kcore
0x7fffc0a3a377 k [k]
trace_event_type_funcs_svcrdma_dma_map_rwctx+0x7fffff3a5eab
0.65% /proc/kcore
0x7fffc0a3a11e k [k] print_fmt_xprtrdma_connect_class+0x7fffff3a5ef2
Not much has changed. Would a full flamegraph/call graph help?
The filesystem setup is 4x NVME in BTRFS RAID 10
root@dbshard1a.prod:~ # btrfs filesystem usage /sqldata
Overall:
Device size: 13.97TiB
Device allocated: 4.04TiB
Device unallocated: 9.93TiB
Device missing: 0.00B
Used: 3.42TiB
Free (estimated): 5.20TiB (min: 5.20TiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data,RAID10: Size:1.79TiB, Used:1.56TiB
/dev/nvme0n1 917.00GiB
/dev/nvme1n1 917.00GiB
/dev/nvme2n1 917.00GiB
/dev/nvme3n1 917.00GiB
Metadata,RAID10: Size:237.00GiB, Used:158.95GiB
/dev/nvme0n1 118.50GiB
/dev/nvme1n1 118.50GiB
/dev/nvme2n1 118.50GiB
/dev/nvme3n1 118.50GiB
System,RAID10: Size:16.00MiB, Used:272.00KiB
/dev/nvme0n1 8.00MiB
/dev/nvme1n1 8.00MiB
/dev/nvme2n1 8.00MiB
/dev/nvme3n1 8.00MiB
Unallocated:
/dev/nvme0n1 2.48TiB
/dev/nvme1n1 2.48TiB
/dev/nvme2n1 2.48TiB
/dev/nvme3n1 2.48TiB