On 12/30/21 20:07, Carlos E. R. wrote:
It fills that 16M of the buffer in dd, and then sends it forward to the pipe. A better processing would start reading another 16M buffer.
So, I wonder if there is something that could be done.
To avoid the read-from-file-and-write-to-stdout in dd, you could try to completely avoid it, and instead to associate /dev/XXX directly as stdin to tee(1). mkfifo mdpipe - dd if=/dev/$1 status=progress bs=16M | tee mdpipe | pigz > $3.gz & + tee mdpipe < /dev/$1 | pigz > $3.gz & # Try using "--fast" next time md5sum -b mdpipe | tee -a md5checksum_expanded Not sure if that's much better in regards of performance, though, but obviously that saves 1x pass of reading + 1x pass of writing of the whole stuff. And of course you won't get the 'progress' output from dd. ;-) FWIW to avoid the fifo file, you could use the process substitution operator >(...) which newer shells support. Examples can be found in the tee documentation: https://www.gnu.org/software/coreutils/tee Finally, you can also play with another compression algorithm/tool than pigz. E.g. zstd seems to be quite fast, and better in compression: The following with pigz takes 2min49s here with sdb1 being a 20G file system (which is 80% used), and the resulting GZ file has 6.2G: $ tee < /dev/sdb1 tee >( pigz > sdb1.gz ) | md5sum > sdb1.md5 while using zstd tool runs 1min17s and results in a 6.0G file: $ tee < /dev/sdb1 tee >( zstd > sdb1.gz ) | md5sum > sdb1.md5 and using 'zstd --fast' has finished after 57s. BTW: I haven't read the complete thread, but I think we can assume that the partition you're copying is not (or at least read-only) mounted, right? Have a nice day, Berny