[opensuse] BASH help - Howto build command line containing pipes and then execute all at once?
Guys (DNH), I'm having trouble building a command line/string containing pipes and then getting it to execute all at once. This is in the lynxdump script I started: http://www.3111skyline.com/dl/dev/scr/net/lynxdump.sh I have created a god-awful nested 'if' jungle in the getdump() function that decides what to do based on command line flag settings. What I want to do is be able to piece together a command line that will ultimately look similar to this depending on the options entered on the command line: lynx -dump "$url" | grep -e '^.*rpm$' | grep -v 'debugsource' | grep -v 'debuginfo' | grep http | sed -e 's/^.*\shttp/http/' >> $ofile Basically what I want to do is respond to the following command line options and add the corresponding statement if the option is given: -r|--rpm grep -e '^.*rpm$' --nodebug grep -v 'debugsource' | grep -v 'debuginfo' I have tried replacing getdump() by simply adding the pieces like this: [[ $flagDir -eq 1 ]] && dirSTR="" || dirSTR=" -e '/\/$/d'" [[ $flagRPM -eq 1 ]] && rpmSTR="| grep -e '^.*rpm$'" || rpmSTR="" [[ $noDebug -eq 1 ]] && debugSTR=" | grep -v 'debugsource' | grep -v 'debuginfo' " || \ debugSTR="" dumpSTR="lynx -dump $url ${rpmSTR}${debugSTR}| grep http | sed -e 's/^.*\shttp/http/'${dirSTR}" Which gives the perfectly correct command line (rpm and nodebug options): lynx -dump http://madwifi-project.org/suse/11.0/x86_64 | grep -e '^.*rpm$' | grep -v 'debugsource' | grep -v 'debuginfo' | grep http | sed -e 's/^.*\shttp/http/' -e '/\/$/d' But when attempting to execute 'dumpSTR' within the script, I end up with an invalid option error from lynx: lynx: Invalid Option: -e Basically, lynx is considering everything in the string as options to lynx instead of recognizing the pipe. I have checked all lynx options hoping for a '--' option to say end of options, but no joy. Is there a trick that will work here. I'm sure there is, but I just don't understand it yet. What say the gurus? As last resort, I know I can apply the filters successively to the result of lynx -dump ... by making separate call for each piece, but I wanted to do it all in the single command line like I can just enter on the command line. Is this a stumbling block withing bash? Thanks for any help :) -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2010-06-16 19:39, David C. Rankin wrote:
Guys (DNH),
...
dumpSTR="lynx -dump $url ${rpmSTR}${debugSTR}| grep http | sed -e 's/^.*\shttp/http/'${dirSTR}"
Which gives the perfectly correct command line (rpm and nodebug options):
lynx -dump http://madwifi-project.org/suse/11.0/x86_64 | grep -e '^.*rpm$' | grep -v 'debugsource' | grep -v 'debuginfo' | grep http | sed -e 's/^.*\shttp/http/' -e '/\/$/d'
But when attempting to execute 'dumpSTR' within the script, I end up with an invalid option error from lynx:
lynx: Invalid Option: -e
Basically, lynx is considering everything in the string as options to lynx instead of recognizing the pipe. I have checked all lynx options hoping for a '--' option to say end of options, but no joy. Is there a trick that will work here. I'm sure there is, but I just don't understand it yet. What say the gurus?
I'm not an expert, I did not understand all your logic on my light read. However, I know this: pipes are interpreted by the shell. If lynx is seeing behind the 1st pipe symbol, it means that bash is not seeing it first. Maybe bash does not intervene in the line execution? - -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iF4EAREIAAYFAkwZE1sACgkQja8UbcUWM1wseAD9G/EIa4gcEkLJ1mUDe6b6zmJK Ei0qmLoZz6scUw9c/yYA/RFUfDmBOPkaahkwDHjoNCrbtN7hAGxxRcni48kaYMr9 =YUWG -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 06/16/2010 01:09 PM, Carlos E. R. wrote:
I'm not an expert, I did not understand all your logic on my light read. However, I know this: pipes are interpreted by the shell. If lynx is seeing behind the 1st pipe symbol, it means that bash is not seeing it first. Maybe bash does not intervene in the line execution?
Carlos, You are right on-the-money with your thinking. It looks like if the pipes are contained in a variable expression, then they are not seen at all and lynx just happily reads past them until it hits an '-e' and pukes. I guess the problem is that I can't put an explicit pipe in unless I break the expression up something like this: dumpSTR="lynx -dump $url ${rpmSTR} | ${debugSTR_1} | ${debugSTR} | grep http | sed -e 's/^.*\shttp/http/'${dirSTR}" but then you have the problem of $rpmSTR being undefined or "" if the option isn't set and that results in either: | | - bash: syntax error near unexpected token `|'; or | "" | - bash: : command not found I guess I would need some type of 'stub' function that read stdin and then wrote to stdout. I'll keep my 'if' jungle at that point :p -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hello, On Wed, 16 Jun 2010, David C. Rankin wrote:
Guys (DNH),
;) [..]
decides what to do based on command line flag settings. What I want to do is be able to piece together a command line that will ultimately look similar to this depending on the options entered on the command line:
lynx -dump "$url" | grep -e '^.*rpm$' | grep -v 'debugsource' | grep -v 'debuginfo' | grep http | sed -e 's/^.*\shttp/http/' >> $ofile [..] I have tried replacing getdump() by simply adding the pieces like this:
[[ $flagDir -eq 1 ]] && dirSTR="" || dirSTR=" -e '/\/$/d'" [[ $flagRPM -eq 1 ]] && rpmSTR="| grep -e '^.*rpm$'" || rpmSTR="" [[ $noDebug -eq 1 ]] && debugSTR=" | grep -v 'debugsource' | grep -v 'debuginfo' " || \ debugSTR=""
dumpSTR="lynx -dump $url ${rpmSTR}${debugSTR}| grep http | sed -e 's/^.*\shttp/http/'${dirSTR}"
Which gives the perfectly correct command line (rpm and nodebug options): [..] Basically, lynx is considering everything in the string as options to lynx instead of recognizing the pipe.
lynx has nothing to do with that. Save the dump and try with 'cat dumpfile ...'. It's just how bash parses/expands/executes commandlines. Pipes (and process substitution) are done first. Only after that, variables are expanded. Arrays too. Using 'set -x' helps, as so often: $ dump=( "cat" "lx.dump" "|" "grep" "-e" '^.*rpm$' ) $ set -x $ ${dump[@]} > /dev/null + cat lx.dump '|' grep -e '^.*rpm$' cat: |: No such file or directory cat: grep: No such file or directory cat: ^.*rpm$: No such file or directory Your only chance is 'eval', and that is esp. ugly.
As last resort, I know I can apply the filters successively to the result of lynx -dump ... by making separate call for each piece, but I wanted to do it all in the single command line like I can just enter on the command line. Is this a stumbling block withing bash?
See above. You could use something like this: filter() { { if test $flagRPM -eq 1 ; then grep -e '^.*rpm$' else cat fi } | { if test $flagDir -eq 1 ; then sed -e '/\/$/d' else cat fi } | { if test $noDebug -eq 1 ; then grep -v 'debugsource\|debuginfo' else cat fi } } But that, too, is ugly. How about the following: lynx -dump "$url" | awk -v rpm="$flagRPM" -v dir="$flagDir" \ -v dbg="$noDebug" ' /http/ { if( dir == 1 && /\/$/ ) { next; } if( rpm == 1 && $0 !~ /^.*rpm$/ ) { next; } if( dbg == 1 && /debug(source|info)/ ) { next; } # sub(/^.*\s*http/, "http"); # print; print $2; }' Remember: always quote your variables (here: $url) ;) Anyway, awk should be much faster too, as it's only one process for all the filtering. And this: grep http | sed -e 's/^.*\shttp/http/' is a useless use of grep. Use sed -e '/http/s/^.*\shttp/http/;' instead. And '\s' is a newish GNUism, AFAIK. HTH, -dnh PS: There's more in that script, haven't gotten around to it. I'd use perl for stuff like this anyway. Latest script GETs a request at http://www.airlinecodes.co.uk/aptcoderes.asp with LWP, extract the block I need, dumps the result with w3m (couldn't be arsed to parse the stuff with one of the HTML/XML parsers there are in/for perl), and voila, I got a nicely formatted html-table (courtesy of w3m ;) 42 lines of generously formatted code. You could port above awk almost 1:1 to perl, add the GET and in the case of such index pages just parse the raw html (it's simple enough in the madwifi case). Or use lynx -dump as well. -- we are apt of borg - rpm is futile - you will be dpkg'ed. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Jun 16, 2010 at 1:39 PM, David C. Rankin
Guys (DNH),
I'm having trouble building a command line/string containing pipes and then getting it to execute all at once. This is in the lynxdump script I started:
http://www.3111skyline.com/dl/dev/scr/net/lynxdump.sh
I have created a god-awful nested 'if' jungle in the getdump() function that decides what to do based on command line flag settings. What I want to do is be able to piece together a command line that will ultimately look similar to this depending on the options entered on the command line:
lynx -dump "$url" | grep -e '^.*rpm$' | grep -v 'debugsource' | grep -v 'debuginfo' | grep http | sed -e 's/^.*\shttp/http/' >> $ofile
Basically what I want to do is respond to the following command line options and add the corresponding statement if the option is given:
-r|--rpm grep -e '^.*rpm$' --nodebug grep -v 'debugsource' | grep -v 'debuginfo'
I have tried replacing getdump() by simply adding the pieces like this:
[[ $flagDir -eq 1 ]] && dirSTR="" || dirSTR=" -e '/\/$/d'" [[ $flagRPM -eq 1 ]] && rpmSTR="| grep -e '^.*rpm$'" || rpmSTR="" [[ $noDebug -eq 1 ]] && debugSTR=" | grep -v 'debugsource' | grep -v 'debuginfo' " || \ debugSTR=""
dumpSTR="lynx -dump $url ${rpmSTR}${debugSTR}| grep http | sed -e 's/^.*\shttp/http/'${dirSTR}"
Which gives the perfectly correct command line (rpm and nodebug options):
lynx -dump http://madwifi-project.org/suse/11.0/x86_64 | grep -e '^.*rpm$' | grep -v 'debugsource' | grep -v 'debuginfo' | grep http | sed -e 's/^.*\shttp/http/' -e '/\/$/d'
But when attempting to execute 'dumpSTR' within the script, I end up with an invalid option error from lynx:
lynx: Invalid Option: -e
Basically, lynx is considering everything in the string as options to lynx instead of recognizing the pipe. I have checked all lynx options hoping for a '--' option to say end of options, but no joy. Is there a trick that will work here. I'm sure there is, but I just don't understand it yet. What say the gurus?
As last resort, I know I can apply the filters successively to the result of lynx -dump ... by making separate call for each piece, but I wanted to do it all in the single command line like I can just enter on the command line. Is this a stumbling block withing bash?
Thanks for any help :)
-- David C. Rankin, J.D.,P.E.
David, I would think you could use a subshell to do it, but for some reason I can't get even the simple command below to work the way I expect: sh -c "ls -l | awk '{ print $2 }' " But, maybe it will prod your thoughts in a good direction. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 06/17/2010 12:01 PM, Greg Freemyer wrote:
David,
I would think you could use a subshell to do it, but for some reason I can't get even the simple command below to work the way I expect:
sh -c "ls -l | awk '{ print $2 }' "
But, maybe it will prod your thoughts in a good direction.
Greg
Thanks Greg, It's the little things that kill me to :p That's why if you look at the bottom of any of my scripts after the 'exit 0', there is always a section labeled: ## Scraps Funny thing is, the scraps section is usually much longer than the finished script itself. I'll give the subshell approach a go. If it works, then all I need to do is wrap my expression in parenthesis and I'm good to go (never works out that easy though) Thank everyone for all your help. I have a bit of digestion of the new stuff to do, but I'll report back with what I come up with. In the mean time if anyone is still running 11.0 (or you have an ATI card of the pre-2400 series design and need to stay with 11.0) give the lynxdump.sh script a go and save the rpms you need to rebuild after a HD crash. For example, I ran it against the 11.0 update repo and got the list: http://www.3111skyline.com/dl/openSUSE_11.0/misc/updtget.bz2 Then simply parsed it with a for loop and grep against my local repo to whittle it down to just a few files I needed: http://www.3111skyline.com/dl/openSUSE_11.0/misc/final wget -i final and my local repo of the 11.0 update files was complete :-) -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (4)
-
Carlos E. R. -
David C. Rankin -
David Haller -
Greg Freemyer