Marcus Meissner said the following on 05/21/2012 05:08 PM:

On Mon, May 21, 2012 at 04:01:16PM -0500, David C. Rankin wrote:

...

...

Bash gurus,

Here is one I don't understand. I needed a simple script to part gtkrc and convert the r,g,b (0.0-1.0) values to (0-255). So I decided to use 'calc' to get the floating point values. The script was:

Did you try that with trace/debug turned on?

From MAN

--debugger Arrange for the debugger profile to be executed before the shell starts. Turns on extended debugging mode (see the description of the extdebug option to the shopt builtin below). and set [+abefhkmnptuvxBCEHPT] [+o option-name] [arg ...] with -v Print shell input lines they are read. -x After expanding each simple command, for command, case command, select command, or arithmetic for command, display the expanded value of PS4, followed by the command and its expanded arguments or associated word list. so as to find out the where and why of ${bg} being dragged in. You may want to make use of --norc Do not read and execute the personal initialization file ~/.bashrc if the shell is interactive. -- A program designed for inputs from people is usually stressed beyond breaking point by computer-generated inputs. -- Dennis Ritchie -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

David Haller said the following on 05/22/2012 06:17 AM:

The 'bg' (resp. 'bag') undefined was produced by _calc_ on the line _r=$(calc -p "255 * $r") on the first round. So, what happens? calc reads stdin after doing the "255 * $r" calculation, reading the next line of the gtkrc, i.e. the line bg[SELECTED] = { 0.216, 0.286, 0.431 } (or in my version: bag[SELECTED] = { 0.216, 0.286, 0.431 })!

Something is wrong here. The script should work if run from cron, from the console CLI, across a telnet/ssh session, as well as under an xterm from any GUI, never mind being invoked from another script or program or used as a library function. Why is gtkrc being read in? -- The most important thing in communication is to hear what isn't being said. --Peter F. Drucker -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Hello, On Tue, 22 May 2012, Anton Aylward wrote:

Anton Aylward said the following on 05/22/2012 08:53 AM:

...

David Haller said the following on 05/22/2012 06:17 AM:

...

The 'bg' (resp. 'bag') undefined was produced by _calc_ on the line _r=$(calc -p "255 * $r") on the first round. So, what happens? calc reads stdin after doing the "255 * $r" calculation, reading the next line of the gtkrc, i.e. the line bg[SELECTED] = { 0.216, 0.286, 0.431 } (or in my version: bag[SELECTED] = { 0.216, 0.286, 0.431 })!

Something is wrong here. The script should work if run from cron, from the console CLI, across a telnet/ssh session, as well as under an xterm from any GUI, never mind being invoked from another script or program or used as a library function.

Why is gtkrc being read in?

Because that's the whole point of the script. Oh, you mean by calc? Because calc 'inherits' the stdin from the while-loop.

Sorry, I worded that badly. I realise that the test file is gtkrc-test.txt, but didn't David try it with other files as well?

No, dcr tried it with a for-loop, where no stdin was "in the loop". In the script though, he fed the while-loop-stdin from the gtkrc and that's the stdin calc read as well. And that's also why the 'echo .. | calc' works, as then calc's stdin is fed by echo and not from the while-loop i.e. the gtkrc. Compare to similar problems using ssh, which even has the option '-n' for exactly that problem. Anyway: redirecting calc's stdin from /dev/null works, as calc then does _not_ read the while-loop-stdin i.e. the gtkrc and barf's on the next line "up", which happens to be the second 'bg[' line (or bag in my modified version). Again: I changed the first uncommented lines that calc get fed in the gtkrc to read: { bug[NORMAL] = { 0.125, 0.129, 0.149 } bag[SELECTED] = { 0.216, 0.286, 0.431 } bg[INSENSITIVE] = { 0.125, 0.129, 0.149 } So, what happens? dcr's script reads the file up to 'bug', parses that to the variables $r, $g, $b and then calls _r=$(calc -p "255 * $r") Where is stdin of the loop at this point? The line 'bug[...' was read, nextup in stdin is 'bag[...]'. And that's what calc reads, tries to evaluate and thus "barfs" "'bag' undefined".. Confused yet? HTH, -dnh -- ``I think I recently decided that if pain is the body's way of saying "Wow, shit, stop! Something is wrong here!", then painkillers are our way of saying "Lalalala! I can't hear you!"'' -- Simon Cozens -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Hello, On Tue, 22 May 2012, David C. Rankin wrote:

On 05/22/2012 08:56 AM, David Haller wrote:

...

Confused yet?

Yes - thoroughly...

Digesting the wisdom, it seems that there is a difference in how calc is treating stdin depending on whether it is called from within a while loop or otherwise.

Nope, calc seems to _always_ read stdin.

The bug/bag example was great exposing what calc was choking on. But I can't for the life of me see how calc can inherit stdin differently when called within a while loop or for loop.

inside a 'while ...; done

I see in the for loop there is no stdin to inherit -- I get that.

Exactly. That's the point why it works in the for-loop.

I also see in the while loop that the potential is there due to redirecting gtkrc-file.txt to feed the while loop -- but that is where things cloud over and my eyes roll back in my head.

See above: inside 'while ... done' all processes share the same stdin unless stdin is redirected. The read immediately after while, inside the loop or any other program. Compare: $ printf "bar %s\n" $(seq 8) | while read x; read y; do \ read z; echo "x=$x; y=$y; z=$z"; \ done x=bar 1; y=bar 2; z=bar 3 x=bar 4; y=bar 5; z=bar 6 x=bar 7; y=bar 8; z=

Where the train wreck occurs in my mind is that regardless of while or for, the line is passed into variable 'l' which is parsed into 'rgb' and then into 'r', 'g' & 'b' --before-- being fed to calc. How does calc even know of a different stdin? Passing '

Nope, 'read l' and calc _both_ read the same stdin, as demonstrated above. UNLESS you redirect calc's stdin, it while also read stdin and thus read the file. Again: $ printf "bar %s\n" $(seq 8) | while read x; read y; do \ z=$(calc -p '21+21'); echo "x=$x; y=$y; z=$z"; \ done x=bar 1; y=bar 2; z=bar 3 x=bar 4; y=bar 5; z=bar 6 x=bar 7; y=bar 8; z= $ printf "bar %s\n" $(seq 8) | while read x; read y; do \ z=$(calc -p '21+21'); echo "x=$x; y=$y; z=$z"; \ done "bar" is undefined x=bar 1; y=bar 2; z=42 And stdin 'is empty', calc has read all lines, read x doesn't see any more lines: $ printf "bar %s\n" $(seq 8) | while read x; read y; do \ z=$(strace -s 40 -eread calc -p '21+21'); echo "x=$x; y=$y; z=$z"; \ done [more reads of ELF binaries snipped and pruned] read(3, "\177ELF..., 832) = 832 read(0, "bar 3\nbar 4\nbar 5\nbar 6\nbar 7\nbar 8\n", 4096) = 36 "bar" is undefined x=bar 1; y=bar 2; z=42 As you can see in the last 'read', calc read 'fd 0', i.e. stdin upto and _including_ the last line containing 'bar 8\n'. Tries to parse that stuff and barfs on the 'bar 3' on line 3 ('read x' and 'read y' read lines 1 and 2, as you can see in the 'echo' output).

Just to really confuse myself, I decided to re-write the script using a for loop instead of a while loop (I prefer while loops anyway because they will always read the last line of the file...) So:

#!/bin/bash

IFS=$'\n'

for l in $(

With the IFS, that should work. It'll break if IFS you ever forget to set IFS ;) [..]

_r=$(calc -p "255 * $r") _g=$(calc -p "255 * $g") _b=$(calc -p "255 * $b")

Again: you could also use bc (that read _either_ stdin or it's arguments ;), redirect calc's stdin or feed calc's stdin yourself using echo ('_r=$(echo "255 * $r" | calc -p)') etc.

It works without a hitch! So, I learned something new -- even though I'm not sure I understand what I learned. BASH handles while and for differently somehow regarding the inheritance of stdin.

With 'for l in $(< ... ); do' there is no stdin redirected, stdin is the stdin of the script, and calc will _still_ read that! The part inside the '{ }' replaces your script! $ printf "bar %s\n" $(seq 4) | { \ IFS=$'\n'; \ for x in $(seq 10 14); do \ z=$(calc -p "$x+1"); \ echo "x=$x; z=$z"; \ done; } "bar" is undefined x=10; z=11 x=11; z=12 x=12; z=13 x=13; z=14 x=14; z=15 But: the for-loop doesn't abort once calc has read stdin (i.e. the 'printf 'bar %s\n' $(seq 4)' output), as the for loop does not depend on stdin.

I can prove to myself that there is indeed this difference, but other than stumbling across it on a case-by-case basis, I'm certain I still do not understand a 'rule' that tells me I when should look out for this type of problem before I run into it... :)

You should have a look what reads stdin. Using calc, you probably should do this in your script: #!/bin/bash exec

Ouch -- my head hurts... dnh, all, I tip my hats to you in your ability to step through and debug this. That isn't something I've mastered yet. Thanks.

Well, it's a lot of basic(s) understanding, years of experience and not being afraid use set -x/set -v, strace and ltrace and wade through their output. Oh, and if neccessary, to have a look into the sources ;) Coming up with a solution is often not that difficult, but understanding _why_ something happens, what's the cause and the correct fix of something, that's taking the extra work. As evidenced in this thread. There were fast a couple of workarounds. But I hope in my mails I could explain cause and effects (and how you can diagnose that stuff). Still confused? Play around with my samples a bit (replacing my seq stuff by files if you care). Exursion: Play around with 'strace'. Maybe use strace -f -s 128 -eread,file,process -o yourscript.strace bash ./yourscript and look for strings from your input file in that strace, like that 'read(0, ...)' above, look from what process id that read is, search back to a 'fork' ... For example, using the second to last example without the exec So, pid 32143 is 'calc' reading my stdin ;) Thus I _know_ that calc has gobbled up the printf+seq 4 output from stdin ... In the normal '-e process' strace, you'll have either a 'fork' or a 'clone' call a couple of lines before the 'execve', that's where the process is created, the 'execve' then starts it. Following that, I find the next call of calc (execve): [pid 32145] execve("/usr/bin/calc", ["calc", "-p", "11+1"], [/* 131 vars */]) = 0 [pid 32145] read(0, "", 4096) = 0 Process 32144 resumed Again: proof that calc reads stdin, no matter what. Nassssty calc! ;) HTH, and ask if you're still (or again) confused, -dnh [1] I wonder how this might gets people using a search-engine to find this mail ;) -- $SUPPLIER said "next day delivery". Unfortunately, they didn't specify which day it would be next to. -- James Cort -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Hello, On Thu, 24 May 2012, Anton Aylward wrote:

David Haller said the following on 05/24/2012 06:57 AM:

...

Nope, calc seems to _always_ read stdin.

AH! *THAT* Answers my previous (badly worded) question.

Hm. I thought I'd written that (and not too unclear), or was that to me too clear to state that explicitly enough? But, at least now I seem to have cleared up that fact ;) Oh, and BTW, I've seen no option to 'calc' to change that behaviour. But I guess I've pointed out enough options how to remedy that in a script (calc

Hello, On Thu, 24 May 2012, David C. Rankin wrote:

On 05/24/2012 07:20 AM, David Haller wrote:

...

Could be worth a bugreport/feature request upstream. David C.R., mail me if you don't want to do that yourself. Oh, and BTW@dcr: please notify me via PM/OBS/Bugzilla if you notice a new upstream version, I don't usually check upstream for new versions more than once in a couple of months (at least such stable stuff as calc seems to be).

I'll file it tonight and cobble together the issue and the situation it solves from this thread. Last I looked it was the original author that still maintains calc. I think I built calc from source for 11.4 IIRC and we talked about packaging it a bit. Yep:

15:31 alchemy:~> rpm -q calc package calc is not installed 15:31 alchemy:~> calc C-style arbitrary precision calculator (version 2.12.4.4) Calc is open software. For license details type: help copyright [Type "exit" to exit, or "help" for help.]

; quit

That'll give me the info I need :)

Weird. I was quite sure it was you that I cobbled together the RPM for, see[1], seems still up-to-date BTW. -dnh [1] http://download.opensuse.org/repositories/home:/dnh/openSUSE_11.4/ -- Every time I hear someone whining about vi or emacs, I give them ed to play with. Shuts them up PDQ. -- stevo -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Hello, On Fri, 25 May 2012, Patrick Shanahan wrote:

* David Haller [05-25-12 20:27]:

...

On Thu, 24 May 2012, David C. Rankin wrote: ...

...

That'll give me the info I need :)

Weird. I was quite sure it was you that I cobbled together the RPM for, see[1], seems still up-to-date BTW.

May have been for me but I do not remember, old timer's I guess :^)

Uhm, I don't remember. But for what does one have a complete local mail-archive? Ah, found it (with ",/ ~P ~b calc" (i.e. for reverse search on mails by me containing 'calc' in the body)), it's even in the still current mbox :) And it actually _was_ dcr, and that mail doesn't qualify for "oldtimer" (but dcr, you and me do): ==== Date: Thu, 8 Sep 2011 06:11:03 +0200 From: David Haller Subject: Re: [opensuse] Does 'calc' exist as an 11.4 rpm somewhere?? (the CLI calc not office calc) Message-ID: <20110908041103.GA4162@grusum.endjinn.de> [..] On Tue, 06 Sep 2011, David C. Rankin wrote:

Anybody packaged 'calc' as an rpm for 11.4 somewhere. Search on opensuse didn't turn it up. The source is:

http://www.isthe.com/chongo/src/calc/

Please test: http://download.opensuse.org/repositories/home:/dnh [..] ==== IIRC, that was a typical "dnh", grab source, hack up a .spec, and off you go ;)

I have been using calc for several years, used to dl the source and make local version with chkinstall. Mathomatic now seems to be quite capable in similar circumstances, but syntax is different.

From the description in software.o.o mathomatic is quite a bit more

You did follow up on above mail from me though (that probably was what triggered your memory): ==== thanks for handling the calc package. I was using an older version but recently tend to use mathomatic. ==== than a calculator such as dc, bc or calc. Myself? I use my brain, bc, or pen and paper, in that order ;) -dnh -- < "if (((foo == true) == true) == true)..." -- Simon Cozens > -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 05/24/2012 05:57 AM, David Haller wrote:

And stdin 'is empty', calc has read all lines, read x doesn't see any more lines:

So basically in z=$(calc -p '21+21') calc sees: $ calc -p '21+21' \ bar 3 \ bar 4 \ bar 5 \ bar 6 \ bar 7 \ bar 8 and discards everything else in the line beginning with the first 'bar' because as far as calc knows that is some undefined text? But then just continues reading the rest of the 'bar #' until it runs out of things to read?

$ printf "bar %s\n" $(seq 8) | while read x; read y; do \ z=$(strace -s 40 -eread calc -p '21+21'); echo "x=$x; y=$y; z=$z"; \ done [more reads of ELF binaries snipped and pruned] read(3, "\177ELF..., 832) = 832 read(0, "bar 3\nbar 4\nbar 5\nbar 6\nbar 7\nbar 8\n", 4096) = 36 "bar" is undefined

x=bar 1; y=bar 2; z=42

As you can see in the last 'read', calc read 'fd 0', i.e. stdin upto and_including_ the last line containing 'bar 8\n'. Tries to parse that stuff and barfs on the 'bar 3' on line 3 ('read x' and 'read y' read lines 1 and 2, as you can see in the 'echo' output).

Aah.. Gotcha :)

The part inside the '{ }' replaces your script!

$ printf "bar %s\n" $(seq 4) | { \ IFS=$'\n'; \ for x in $(seq 10 14); do \ z=$(calc -p "$x+1"); \ echo "x=$x; z=$z"; \ done; } "bar" is undefined

x=10; z=11 x=11; z=12 x=12; z=13 x=13; z=14 x=14; z=15

But: the for-loop doesn't abort once calc has read stdin (i.e. the 'printf 'bar %s\n' $(seq 4)' output), as the for loop does not depend on stdin.

OK, if I'm learning anything here, then that looks like: $ bar 1 \ bar 2 \ bar 3 \ bar 4 | { ..stuff.. z=$(calc -p "$x+1") ..stuff.. } calc still chokes on 'bar 1', but it has already executed the for x in $(seq 10 14) before it dies. So the calculations finished in the '{}'s before calc ate: 'bar 1 \n bar 2 \n bar 3 \n bar 4'

#!/bin/bash exec

^^^^^^^^^^^^^^^^

IFS=$'\n' for l in $(

Compare:

$ printf "bar %s\n" $(seq 4) | { exec

That way, you redirect your script's stdin from /dev/null, and thus 'calc' can't read anything from there that might confuse it.

That is the 'nugget' to be found. I would not have seen that or recognized that as an option for eons. That's where the (knowledge, skill, training, years of experience, etc..) really makes the difference. I completely lacked the appreciation for that level of redirection understanding. Oh, I've dug fairly far into redirection of stdin, stdout, and stderr, but stumbled face-first into what calc was doing with it ;-) <snip>

HTH, and ask if you're still (or again) confused, -dnh

dnh -- It does! That was excellent, and the archives will hold onto this thread to help close the gap for many to come in the levels of redirection understanding (as well as provide a jumping off point for furthering debugging skills) This is something that probably needs to be forwarded to the ABS folks for inclusion in the section of redirection or as it's own separate 'Advanced Topics in Redirection - Understanding Potential Errors'. What was it -- I believe attributed to Einstein -- "True Genius is the ability to explain the complexities of nature to a child." Thanks! -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

David Haller wrote:

Hello,

On Tue, 22 May 2012, Anton Aylward wrote:

...

Anton Aylward said the following on 05/22/2012 08:53 AM:

...

David Haller said the following on 05/22/2012 06:17 AM:

...

The 'bg' (resp. 'bag') undefined was produced by _calc_ on the line _r=$(calc -p "255 * $r") on the first round. So, what happens? calc reads stdin after doing the "255 * $r" calculation, reading the next line of the gtkrc, i.e. the line bg[SELECTED] = { 0.216, 0.286, 0.431 } (or in my version: bag[SELECTED] = { 0.216, 0.286, 0.431 })!

Something is wrong here. The script should work if run from cron, from the console CLI, across a telnet/ssh session, as well as under an xterm from any GUI, never mind being invoked from another script or program or used as a library function.

Why is gtkrc being read in?

Because that's the whole point of the script. Oh, you mean by calc? Because calc 'inherits' the stdin from the while-loop.

...

Sorry, I worded that badly. I realise that the test file is gtkrc-test.txt, but didn't David try it with other files as well?

No, dcr tried it with a for-loop, where no stdin was "in the loop".

In the script though, he fed the while-loop-stdin from the gtkrc and that's the stdin calc read as well. And that's also why the 'echo .. | calc' works, as then calc's stdin is fed by echo and not from the while-loop i.e. the gtkrc.

Compare to similar problems using ssh, which even has the option '-n' for exactly that problem. Anyway: redirecting calc's stdin from /dev/null works, as calc then does _not_ read the while-loop-stdin i.e. the gtkrc and barf's on the next line "up", which happens to be the second 'bg[' line (or bag in my modified version). Again: I changed the first uncommented lines that calc get fed in the gtkrc to read:

{ bug[NORMAL] = { 0.125, 0.129, 0.149 } bag[SELECTED] = { 0.216, 0.286, 0.431 } bg[INSENSITIVE] = { 0.125, 0.129, 0.149 }

So, what happens? dcr's script reads the file up to 'bug', parses that to the variables $r, $g, $b and then calls

_r=$(calc -p "255 * $r")

Where is stdin of the loop at this point? The line 'bug[...' was read, nextup in stdin is 'bag[...]'. And that's what calc reads, tries to evaluate and thus "barfs" "'bag' undefined"..

barf bag? Is programming getting that bad these days? :-)

Confused yet?

HTH, -dnh

-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 05/22/2012 02:28 PM, Dirk Gently wrote:

why are you running sh filename.sh

when the first line of filename contains this interpreter definition: #!/bin/bash

should be bash filename.sh (but I usually just save the 2 chars.. :) Seriously, In nearly 15 years of this, I can count on one hand and still have fingers left the number of times 'sh foo.sh' has caused issues where 'bash foo.sh' did not. I guess the habit of 'sh scriptname.sh' just comes from the thousands of examples read where 'sh foo.sh' is given. In all the script examples I've seen, I think there has only been 1 or 2 examples that have shown using 'bash foo.sh'. I guess the example writers are human and lazy too. The only place I generally explicitly call bash is when rebooting to recover/replace a root password using: init=/bin/bash in the kernel line in grub. I guess I should make a point of using the interpreter specified after the #! in the future :) -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 05/22/2012 04:00 PM, Dirk Gently wrote:

Once you set the executable bit on a script with an interpreter defined on the first line with the #!/path/to/interpreter mechanism, there's no reason to write bash foo nor python parrot for that file ever again.

I ALWAYS chmod my shell scripts, and then just use them as if it were any other executable.

Yes, You are correct. For the scripts I use regularly, I make them executable and create softlinks to /usr/local/bin and insure it is in my path. But for test scripts, etc.. I admit, I generally just call sh 'whatever' until I put them somewhere for permanent use... /usr/local/bin, crontab, etc.. No rhyme or reason to it other than that. -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org