Comment # 4 on bug 1198972 from Ian Christian

Further to this, it can be reduced down to just the send, taking out of scope
the SSH connection.

Again it's observed that the first chunk of data moves quickly - 34s in and
806M has been written:

 806MiB 0:00:34 [93.9KiB/s] 


However, then things then slow down significantly, while there are occasional
bursts of speed, generally speaking the speed can be sene in 100-200KiB/s

After 4 minutes, only 200MiB of extra data has been written:

 989MiB 0:04:04 [ 144KiB/s] 


Here's the details on the devices:

root@dbshard1a.prod:~ # btrfs device stats /sqldata/
[/dev/nvme0n1].write_io_errs    0
[/dev/nvme0n1].read_io_errs     0
[/dev/nvme0n1].flush_io_errs    0
[/dev/nvme0n1].corruption_errs  0
[/dev/nvme0n1].generation_errs  0
[/dev/nvme1n1].write_io_errs    0
[/dev/nvme1n1].read_io_errs     0
[/dev/nvme1n1].flush_io_errs    0
[/dev/nvme1n1].corruption_errs  0
[/dev/nvme1n1].generation_errs  0
[/dev/nvme2n1].write_io_errs    0
[/dev/nvme2n1].read_io_errs     0
[/dev/nvme2n1].flush_io_errs    0
[/dev/nvme2n1].corruption_errs  0
[/dev/nvme2n1].generation_errs  0
[/dev/nvme3n1].write_io_errs    0
[/dev/nvme3n1].read_io_errs     0
[/dev/nvme3n1].flush_io_errs    0
[/dev/nvme3n1].corruption_errs  0
[/dev/nvme3n1].generation_errs  0
root@dbshard1a.prod:~ # btrfs device usage /sqldata/
/dev/nvme0n1, ID: 1
   Device size:             3.49TiB
   Device slack:              0.00B
   Data,RAID10:           751.00GiB
   Metadata,RAID10:       187.00GiB
   System,RAID10:          32.00MiB
   Unallocated:             2.58TiB

/dev/nvme1n1, ID: 2
   Device size:             3.49TiB
   Device slack:              0.00B
   Data,RAID10:           751.00GiB
   Metadata,RAID10:       187.00GiB
   System,RAID10:          32.00MiB
   Unallocated:             2.58TiB

/dev/nvme2n1, ID: 3
   Device size:             3.49TiB
   Device slack:              0.00B
   Data,RAID10:           751.00GiB
   Metadata,RAID10:       187.00GiB
   System,RAID10:          32.00MiB
   Unallocated:             2.58TiB

/dev/nvme3n1, ID: 4
   Device size:             3.49TiB
   Device slack:              0.00B
   Data,RAID10:           751.00GiB
   Metadata,RAID10:       187.00GiB
   System,RAID10:          32.00MiB
   Unallocated:             2.58TiB



Any suggestions on how to diagnose this speed issue would be much appreciated!