[opensuse] piping question
Hi, I have this line to image a disk partition and compress it: dd if=/dev/nvme0n1p2 status=progress bs=16M | pigz > nvme0n1p2.gz How could I insert in the pipe the md5 calculation of the non-compressed stream, stored to a file? Is it possible? I know I can use "tee" to pipe and create a file, but that is not it. -- Cheers Carlos E. R. (from oS 15.1) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Quoting Carlos E. R.
Hi,
I have this line to image a disk partition and compress it:
dd if=/dev/nvme0n1p2 status=progress bs=16M | pigz > nvme0n1p2.gz
How could I insert in the pipe the md5 calculation of the non-compressed stream, stored to a file? Is it possible?
I can't see any way besides an additional line/command: dd if=/dev/nvme0n1p2 | md5sum > nvm20n1p2.md5 Jeffrey -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
El 2020-09-07 a las 13:45 -0500, Jeffrey L. Taylor escribió:
Quoting Carlos E. R. <>:
Hi,
I have this line to image a disk partition and compress it:
dd if=/dev/nvme0n1p2 status=progress bs=16M | pigz > nvme0n1p2.gz
How could I insert in the pipe the md5 calculation of the non-compressed stream, stored to a file? Is it possible?
I can't see any way besides an additional line/command:
dd if=/dev/nvme0n1p2 | md5sum > nvm20n1p2.md5
Sure, an additional command is how I'm doing it. -- Cheers Carlos E. R. (from openSUSE Leap 15.1 x86_64 (Minas Tirith))
It's slightly different from your command but it does the job and it's a one-liner. dd if=/dev/nvme0n1p2 | tee -a /proc/self/fd/2 2> >(pigz > nvme0n1p2.gz) > > (md5sum > nvme0n1p2.md5) -- /bengan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
W dniu 07.09.2020 o 23:13, Bengt Gördén pisze:
It's slightly different from your command but it does the job and it's a one-liner.
dd if=/dev/nvme0n1p2 | tee -a /proc/self/fd/2 2> >(pigz > nvme0n1p2.gz) > > (md5sum > nvme0n1p2.md5)
Even better: dd if=/dev/nvme0n1p2 | tee -a >(md5sum > nvme0n1p2.md5) | pigz > nvme0n1p2.gz
On 08/09/2020 08.56, Adam Mizerski wrote:
W dniu 07.09.2020 o 23:13, Bengt Gördén pisze:
It's slightly different from your command but it does the job and it's a one-liner.
dd if=/dev/nvme0n1p2 | tee -a /proc/self/fd/2 2> >(pigz > nvme0n1p2.gz) > > (md5sum > nvme0n1p2.md5)
Even better:
dd if=/dev/nvme0n1p2 | tee -a >(md5sum > nvme0n1p2.md5) | pigz > nvme0n1p2.gz
Something like this is what I was thinking of initially but didn't know how to do. :-) What confuses me is the tee. "tee -a" sends to a file, but your are doing a second redirection. I will have to test this later. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
On 2020-09-08 10:01, Carlos E. R. wrote:
On 08/09/2020 08.56, Adam Mizerski wrote:
Even better:
dd if=/dev/nvme0n1p2 | tee -a >(md5sum > nvme0n1p2.md5) | pigz > nvme0n1p2.gz
Something like this is what I was thinking of initially but didn't know how to do. :-)
What confuses me is the tee. "tee -a" sends to a file, but your are doing a second redirection. I will have to test this later.
I was looking for something like this a couple years ago and found it on askubuntu. I tend to save all nice commands for later use :) Here's the original and why it's probably better with /proc/self/fd/2 2 in the long run. For your specific case the shorter one is better. https://askubuntu.com/questions/385264/dump-md5-and-sha1-checksums-with-a-si... -- /bengan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 08/09/2020 10.29, Bengt Gördén wrote:
On 2020-09-08 10:01, Carlos E. R. wrote:
On 08/09/2020 08.56, Adam Mizerski wrote:
Even better:
dd if=/dev/nvme0n1p2 | tee -a >(md5sum > nvme0n1p2.md5) | pigz > nvme0n1p2.gz
Something like this is what I was thinking of initially but didn't know how to do. :-)
What confuses me is the tee. "tee -a" sends to a file, but your are doing a second redirection. I will have to test this later.
I was looking for something like this a couple years ago and found it on askubuntu. I tend to save all nice commands for later use :)
Here's the original and why it's probably better with /proc/self/fd/2 2 in the long run. For your specific case the shorter one is better.
https://askubuntu.com/questions/385264/dump-md5-and-sha1-checksums-with-a-si...
Nice :-) (I'll look at the link later, I have firefox "off" now, while I run an rsync live backup) Somehow, I find the implementation with named pipes "cute" :-) -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
On 2020-09-08 10:01, Carlos E. R. wrote:
On 08/09/2020 08.56, Adam Mizerski wrote:
dd if=/dev/nvme0n1p2 | tee -a >(md5sum > nvme0n1p2.md5) | pigz > nvme0n1p2.gz
Something like this is what I was thinking of initially but didn't know how to do. :-)
What confuses me is the tee. "tee -a" sends to a file, but your are doing a second redirection. I will have to test this later.
It's also documented here: https://www.gnu.org/software/coreutils/tee The >(...) syntax is a non-portable shell feature though. The implementation with mkfifo will work everywhere. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tuesday 08 September 2020, Carlos E. R. wrote:
Hi,
I have this line to image a disk partition and compress it:
dd if=/dev/nvme0n1p2 status=progress bs=16M | pigz > nvme0n1p2.gz
How could I insert in the pipe the md5 calculation of the non-compressed stream, stored to a file? Is it possible?
Perhaps you could use a named pipe? mkfifo mdpipe dd if=/dev/nvme0n1p2 status=progress bs=16M | tee mdpipe | pigz > nvme0n1p2.gz & md5sum mdpipe Michael -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
El 2020-09-08 a las 09:24 +1200, Michael Hamilton escribió:
On Tuesday 08 September 2020, Carlos E. R. wrote:
Hi,
I have this line to image a disk partition and compress it:
dd if=/dev/nvme0n1p2 status=progress bs=16M | pigz > nvme0n1p2.gz
How could I insert in the pipe the md5 calculation of the non-compressed stream, stored to a file? Is it possible?
Perhaps you could use a named pipe?
mkfifo mdpipe dd if=/dev/nvme0n1p2 status=progress bs=16M | tee mdpipe | pigz > nvme0n1p2.gz & md5sum mdpipe
That is a curious one, thanks. Also to Bengt Gördén. None of that would have occurred to me. -- Cheers Carlos E. R. (from openSUSE Leap 15.1 x86_64 (Minas Tirith))
El 2020-09-07 a las 23:55 +0200, Carlos E. R. escribió:
El 2020-09-08 a las 09:24 +1200, Michael Hamilton escribió:
On Tuesday 08 September 2020, Carlos E. R. wrote:
Hi,
I have this line to image a disk partition and compress it:
dd if=/dev/nvme0n1p2 status=progress bs=16M | pigz > nvme0n1p2.gz
How could I insert in the pipe the md5 calculation of the non-compressed stream, stored to a file? Is it possible?
Perhaps you could use a named pipe?
mkfifo mdpipe dd if=/dev/nvme0n1p2 status=progress bs=16M | tee mdpipe | pigz > nvme0n1p2.gz & md5sum mdpipe
That is a curious one, thanks. Also to Bengt Gördén. None of that would have occurred to me.
It works :-) I did this test, those are small partitions: #!/bin/bash # timing and verification direct checksum time dd if=/dev/nvme0n1p1 status=progress bs=16M | md5sum -b | tee md5checksum_direct time dd if=/dev/nvme0n1p1 status=progress bs=16M | pigz > nvme0n1p1.gz echo "····" time dd if=/dev/nvme0n1p2 status=progress bs=16M | md5sum -b | tee -a md5checksum_direct time dd if=/dev/nvme0n1p2 status=progress bs=16M | pigz > nvme0n1p2.gz echo "····" time dd if=/dev/nvme0n1p4 status=progress bs=16M | md5sum -b | tee -a md5checksum_direct time dd if=/dev/nvme0n1p4 status=progress bs=16M | pigz > nvme0n1p4.gz echo "····" echo "======" mkfifo mdpipe dd if=/dev/nvme0n1p1 status=progress bs=16M | tee mdpipe | pigz > nvme0n1p1.gz & time md5sum -b mdpipe | tee md5checksum_expanded wait rm mdpipe mkfifo mdpipe dd if=/dev/nvme0n1p2 status=progress bs=16M | tee mdpipe | pigz > nvme0n1p2.gz & time md5sum -b mdpipe | tee -a md5checksum_expanded wait rm mdpipe mkfifo mdpipe dd if=/dev/nvme0n1p4 status=progress bs=16M | tee mdpipe | pigz > nvme0n1p4.gz & time md5sum -b mdpipe | tee -a md5checksum_expanded wait rm mdpipe exit The "direct" batch took: real 0m0,510s user 0m0,400s sys 0m0,167s +real 0m0,791s user 0m4,750s sys 0m0,237s real 0m0,048s user 0m0,025s sys 0m0,018s +real 0m0,126s user 0m0,807s sys 0m0,042s real 0m1,943s user 0m1,635s sys 0m0,512s +real 0m6,016s user 0m43,107s sys 0m1,510s The combined took: real 0m1,103s user 0m0,467s sys 0m0,151s real 0m0,123s user 0m0,025s sys 0m0,039s real 0m6,920s user 0m1,860s sys 0m1,460s On the case of the third partition, which is larger (1G), the direct test takes 7.9s, and the combined takes 6.9s. The combined version is just a bit faster. Considering that the total is 500 GB, there may be a small significative gain. Does the named pipe "live" in ram or in disk? I might push it to a tmpfs in the second case. checksums 1e5cac9ebaf40056963355111a430988 *- 9830a82f1b9b4e9a4691ca79e0e38a25 *- 757da9c4aca933c2069bb8105757763d *- 1e5cac9ebaf40056963355111a430988 *mdpipe 9830a82f1b9b4e9a4691ca79e0e38a25 *mdpipe 757da9c4aca933c2069bb8105757763d *mdpipe They match :-) -- Cheers Carlos E. R. (from openSUSE Leap 15.1 x86_64 (Minas Tirith))
On Tuesday 08 September 2020, Carlos E. R. wrote:
... Does the named pipe "live" in ram or in disk? I might push it to a tmpfs in the second case.
...
As I understand it Linux implements named pipes with buffer memory, they are not backed by disk. The amount of allocated memory is limited, if the written data is not consumed fast enough, the writer will block until the reader catches up - you can test this by not reading from the fifo, the dd will block. At least that how things appear to work when I tested it by using a Tumbleweed 4GB ISO. Michael -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
W dniu 08.09.2020 o 03:40, Michael Hamilton pisze:
On Tuesday 08 September 2020, Carlos E. R. wrote:
... Does the named pipe "live" in ram or in disk? I might push it to a tmpfs in the second case.
...
As I understand it Linux implements named pipes with buffer memory, they are not backed by disk. The amount of allocated memory is limited, if the written data is not consumed fast enough, the writer will block until the reader catches up - you can test this by not reading from the fifo, the dd will block. At least that how things appear to work when I tested it by using a Tumbleweed 4GB ISO.
Michael
That's true. There's a hack that allows to use this to detect on the server if you're using `curl | bash`. If bash is not reading input, then curl pauses downloading. https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/ If you want to have a bigger buffer, you can put a `buffer` command (sudo zypper in buffer) in the pipeline.
On 08/09/2020 09.06, Adam Mizerski wrote:
W dniu 08.09.2020 o 03:40, Michael Hamilton pisze:
On Tuesday 08 September 2020, Carlos E. R. wrote:
... Does the named pipe "live" in ram or in disk? I might push it to a tmpfs in the second case.
...
As I understand it Linux implements named pipes with buffer memory, they are not backed by disk. The amount of allocated memory is limited, if the written data is not consumed fast enough, the writer will block until the reader catches up - you can test this by not reading from the fifo, the dd will block. At least that how things appear to work when I tested it by using a Tumbleweed 4GB ISO.
Perfect! :-)
That's true.
There's a hack that allows to use this to detect on the server if you're using `curl | bash`. If bash is not reading input, then curl pauses downloading. https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/
If you want to have a bigger buffer, you can put a `buffer` command (sudo zypper in buffer) in the pipeline.
Yes, I remember reading about that command long ago. -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
On 2020-09-07 07:35:30 Carlos E. R. wrote:
Hi,
I have this line to image a disk partition and compress it:
dd if=/dev/nvme0n1p2 status=progress bs=16M | pigz > nvme0n1p2.gz
How could I insert in the pipe the md5 calculation of the non-compressed stream, stored to a file? Is it possible?
I know I can use "tee" to pipe and create a file, but that is not it.
-- Cheers
Carlos E. R. (from oS 15.1)
The info file for tee shows how to do this. Leslie -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 08/09/2020 10.49, J Leslie Turriff wrote:
On 2020-09-07 07:35:30 Carlos E. R. wrote:
Hi,
I have this line to image a disk partition and compress it:
dd if=/dev/nvme0n1p2 status=progress bs=16M | pigz > nvme0n1p2.gz
How could I insert in the pipe the md5 calculation of the non-compressed stream, stored to a file? Is it possible?
I know I can use "tee" to pipe and create a file, but that is not it.
The info file for tee shows how to do this.
Does it? Gosh, I read it time ago. [...] Yes, it does :-o # slightly contrived, to demonstrate process substitution wget -O - https://example.com/dvd.iso \ | tee >(sha1sum > dvd.sha1) > dvd.iso ... «Note, however, that this example relies on a feature of modern shells called “process substitution” (the ‘>(command)’ syntax, above; *Note Process Substitution: (bash)Process Substitution.), so it works with ‘zsh’, ‘bash’, and ‘ksh’, but not with ‘/bin/sh’. So if you write code like this in a shell script, be sure to start the script with ‘#!/bin/bash’.» ... You can extend this example to make ‘tee’ write to two processes, computing MD5 and SHA1 checksums in parallel. In this case, process substitution is required: wget -O - https://example.com/dvd.iso \ | tee >(sha1sum > dvd.sha1) \ >(md5sum > dvd.md5) \ > dvd.iso Well, well... -- Cheers / Saludos, Carlos E. R. (from 15.1 x86_64 at Telcontar)
participants (7)
-
Adam Mizerski -
Bengt Gördén -
Bernhard Voelker -
Carlos E. R. -
J Leslie Turriff -
Jeffrey L. Taylor -
Michael Hamilton