[opensuse] shell script newbie: how to display progress of a pipe?
Hello. I have a shell script that process about 35MB data: command1 produce 35MB data and sed do some work to the data, send to command2 who receive these data: $ command1 | sed -e 'xxxx' | command 2 The problem being I don't know the progress. Unfortunately both command1 and command2 do not have any output of progress, maybe 'sed' has a way of displaying progress but I could not find. The commands run very long time (1 day) so I need a way to know how much data has been processed (1MB? 10MB?). I guess maybe there have already been such a useful tool that simply read data from stdin, write to stdout (like cat) but meanwhile produce a message on STDERR saying "xxx bytes has been read". But I couldn't find it. I also think it would be nice if cat(1) or tr(1) have a 'verbose' option that report progress to STDERR, but cat(1) don't seem to have such an option. What would you suggest for my case to report progress? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Tuesday 2007-03-06 at 09:50 +0800, Zhang Weiwu wrote:
Hello. I have a shell script that process about 35MB data: command1 produce 35MB data and sed do some work to the data, send to command2 who receive these data:
$ command1 | sed -e 'xxxx' | command 2
The problem being I don't know the progress. Unfortunately both command1 and command2 do not have any output of progress, maybe 'sed' has a way of displaying progress but I could not find. The commands run very long time (1 day) so I need a way to know how much data has been processed (1MB? 10MB?).
You can do something like this: command1 | tee file1 | sed -e 'xxxx' | tee file 2 | command 2 which will save intermediate results to file1 and file2, and from that you might be able to deduce progress. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFF7M5utTMYHG2NR9URAiWRAJ9KhU93DaxRZD9221lz8hhrUEY1dwCeK/Zm ZMYG7yCvGb5mkjOkA70UnSg= =ip+q -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Weiwu, On Monday 05 March 2007 17:50, Zhang Weiwu wrote:
Hello. I have a shell script that process about 35MB data: command1 produce 35MB data and sed do some work to the data, send to command2 who receive these data:
$ command1 | sed -e 'xxxx' | command 2
The problem being I don't know the progress. Unfortunately both command1 and command2 do not have any output of progress, maybe 'sed' has a way of displaying progress but I could not find. The commands run very long time (1 day) so I need a way to know how much data has been processed (1MB? 10MB?).
Sed can write to outputs other than the standard out. It's a funky language, but you can make sed jump through some hoops if you reall need it to. However, you might want to consider an alternate means of achieving the overall goal (whatever the sed filter did plus the progress reporting) by recasting the task in awk or Perl.
I guess maybe there have already been such a useful tool that simply read data from stdin, write to stdout (like cat) but meanwhile produce a message on STDERR saying "xxx bytes has been read". But I couldn't find it. I also think it would be nice if cat(1) or tr(1) have a 'verbose' option that report progress to STDERR, but cat(1) don't seem to have such an option.
If you configure Guru's RPM repository as a YaST repository and search (in Software Management) on the term "progress", you'll find some tools that may serve your needs: - barcat - 'cat' Tool with an ASCII Progress Bar barcat is just like "cat", but displays an ASCII progress bar. Its goal is to work on any Unix-like platform and to be directly usable in installation scripts without needing compilation, so it is a pure shell script. - clpbar - Command Line Progress Bar Command Line Progress Bar is a simple command line tool to display information about a data transfer stream. It will display the number of bytes transfered, the speed of the transfer, and if the size of the data stream is known it will display the ETA. - pipemeter - Provides Throughput and sometimes Progress on Shell Pipes This program can be used in a shell pipe to display speed and progress (if size of stream is available). - progress - File I/O Progress Monitor Utility The progress utility allows progress to be monitored of file I/O. It includes support for gzip-compressed files, so "progress -z -f file.tar.gz tar xf -" would show progress of extracting file.tar.gz.It's a port of the original NetBSD progress utility to Linux and Solaris. - pv - PipeViewer - Monitor the Progress of Data through Pipes PV ("Pipe Viewer") is a tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.
What would you suggest for my case to report progress?
Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Mon, 2007-03-05 at 18:26 -0800, Randall R Schulz wrote:
If you configure Guru's RPM repository as a YaST repository and search (in Software Management) on the term "progress", you'll find some tools that may serve your needs:
Thanks a lot! This is a good experience on getting me used to search on repository! Thanks. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Monday 05 March 2007 19:50, Zhang Weiwu wrote:
What would you suggest for my case to report progress? Write a non-buffering stage to preceed the sed stage.
command | non-buffering-counter | sed-stage | command2 The non-buffering counter stage immediately passes each input to its output (does not buffer [stack] the data) and also keeps a running total of bytes/words that can be posted to a log file at convenient intervals of mbs. On the other hand... we have an English saying that probably applies in China also... "a watched pot never boils". If you know the commands are reliable then watching their progress (esp if you know it is going to take an entire day) is silly... just wastes memory and slows the overall performance of the process... dude, just let it run. -- Kind regards, M Harris <>< -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 3/6/07, M Harris
On Monday 05 March 2007 19:50, Zhang Weiwu wrote:
What would you suggest for my case to report progress? Write a non-buffering stage to preceed the sed stage.
command | non-buffering-counter | sed-stage | command2
The non-buffering counter stage immediately passes each input to its output (does not buffer [stack] the data) and also keeps a running total of bytes/words that can be posted to a log file at convenient intervals of mbs.
dd might be non-buffering if you set it low-enough blocksize. something like: command | dd bs=1b | sed-stage | command2 Then if you can figure out how to get the pid for dd. (ie. I don't know how to get the pid of a command in the middle of a pipe call like that. Maybe after the fact via ps -ef | grep | cut ...) You can have a loop that tells dd to give a status update every 10 minutes or so: while true do sleep 600 kill -sigusr1 $dd_pid done None of this is tested but you get the idea. Greg -- Greg Freemyer The Norcross Group Forensics for the 21st Century -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (5)
-
Carlos E. R. -
Greg Freemyer -
M Harris -
Randall R Schulz -
Zhang Weiwu