El 2020-09-07 a las 23:55 +0200, Carlos E. R. escribió:
El 2020-09-08 a las 09:24 +1200, Michael Hamilton escribió:
On Tuesday 08 September 2020, Carlos E. R. wrote:
Hi,
I have this line to image a disk partition and compress it:
dd if=/dev/nvme0n1p2 status=progress bs=16M | pigz > nvme0n1p2.gz
How could I insert in the pipe the md5 calculation of the non-compressed stream, stored to a file? Is it possible?
Perhaps you could use a named pipe?
mkfifo mdpipe dd if=/dev/nvme0n1p2 status=progress bs=16M | tee mdpipe | pigz > nvme0n1p2.gz & md5sum mdpipe
That is a curious one, thanks. Also to Bengt Gördén. None of that would have occurred to me.
It works :-) I did this test, those are small partitions: #!/bin/bash # timing and verification direct checksum time dd if=/dev/nvme0n1p1 status=progress bs=16M | md5sum -b | tee md5checksum_direct time dd if=/dev/nvme0n1p1 status=progress bs=16M | pigz > nvme0n1p1.gz echo "····" time dd if=/dev/nvme0n1p2 status=progress bs=16M | md5sum -b | tee -a md5checksum_direct time dd if=/dev/nvme0n1p2 status=progress bs=16M | pigz > nvme0n1p2.gz echo "····" time dd if=/dev/nvme0n1p4 status=progress bs=16M | md5sum -b | tee -a md5checksum_direct time dd if=/dev/nvme0n1p4 status=progress bs=16M | pigz > nvme0n1p4.gz echo "····" echo "======" mkfifo mdpipe dd if=/dev/nvme0n1p1 status=progress bs=16M | tee mdpipe | pigz > nvme0n1p1.gz & time md5sum -b mdpipe | tee md5checksum_expanded wait rm mdpipe mkfifo mdpipe dd if=/dev/nvme0n1p2 status=progress bs=16M | tee mdpipe | pigz > nvme0n1p2.gz & time md5sum -b mdpipe | tee -a md5checksum_expanded wait rm mdpipe mkfifo mdpipe dd if=/dev/nvme0n1p4 status=progress bs=16M | tee mdpipe | pigz > nvme0n1p4.gz & time md5sum -b mdpipe | tee -a md5checksum_expanded wait rm mdpipe exit The "direct" batch took: real 0m0,510s user 0m0,400s sys 0m0,167s +real 0m0,791s user 0m4,750s sys 0m0,237s real 0m0,048s user 0m0,025s sys 0m0,018s +real 0m0,126s user 0m0,807s sys 0m0,042s real 0m1,943s user 0m1,635s sys 0m0,512s +real 0m6,016s user 0m43,107s sys 0m1,510s The combined took: real 0m1,103s user 0m0,467s sys 0m0,151s real 0m0,123s user 0m0,025s sys 0m0,039s real 0m6,920s user 0m1,860s sys 0m1,460s On the case of the third partition, which is larger (1G), the direct test takes 7.9s, and the combined takes 6.9s. The combined version is just a bit faster. Considering that the total is 500 GB, there may be a small significative gain. Does the named pipe "live" in ram or in disk? I might push it to a tmpfs in the second case. checksums 1e5cac9ebaf40056963355111a430988 *- 9830a82f1b9b4e9a4691ca79e0e38a25 *- 757da9c4aca933c2069bb8105757763d *- 1e5cac9ebaf40056963355111a430988 *mdpipe 9830a82f1b9b4e9a4691ca79e0e38a25 *mdpipe 757da9c4aca933c2069bb8105757763d *mdpipe They match :-) -- Cheers Carlos E. R. (from openSUSE Leap 15.1 x86_64 (Minas Tirith))