bash Script Using Filenames with Embedded Spaces
I am trying to write a simple bash script to handle files with embedded spaces in the filename. For example, a listing of a directory using "ls -1" gives: ls -l *.mp3 -rw-r--r-- 1 lucky users 20168832 2006-11-03 17:54 01 Allegro.mp3 -rw-r--r-- 1 lucky users 8105984 2006-11-03 17:53 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 lucky users 10661760 2006-11-03 17:53 03 Rondo - Allegro.mp3 The following simplified script for i in `ls -1 *.mp3` do echo $i done produces the following output: 01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3 I have tried various combinations of single and double quotes to no avail including redirecting the "ls -1" listing to a file and escaping the spaces with a backslash, again to no avail. Surely there must be some trick I can use to handle this anomoly.? Thank you, Lucky Leavell
Lucky Leavell wrote:
I am trying to write a simple bash script to handle files with embedded spaces in the filename. For example, a listing of a directory using "ls -1" gives:
ls -l *.mp3 -rw-r--r-- 1 lucky users 20168832 2006-11-03 17:54 01 Allegro.mp3 -rw-r--r-- 1 lucky users 8105984 2006-11-03 17:53 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 lucky users 10661760 2006-11-03 17:53 03 Rondo - Allegro.mp3
The following simplified script
for i in `ls -1 *.mp3` do echo $i done
produces the following output:
01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3
I have tried various combinations of single and double quotes to no avail including redirecting the "ls -1" listing to a file and escaping the spaces with a backslash, again to no avail.
Surely there must be some trick I can use to handle this anomoly.?
Thank you, Lucky Leavell
Start off your script with IFS="" and you will probably be successul. At least I was when I just tried it a moment ago.
* Lucky Leavell
The following simplified script
for i in `ls -1 *.mp3` do echo $i done
perhaps use 'tr' to translate the 'space' to 'underline' or some other normally unused char in the 'ls ...' and back in the 'echo $i...' ??? -- Patrick Shanahan Registered Linux User #207535 http://wahoo.no-ip.org @ http://counter.li.org HOG # US1244711 Photo Album: http://wahoo.no-ip.org/gallery2
On Monday 06 November 2006 14:26, Lucky Leavell wrote:
I am trying to write a simple bash script to handle files with embedded spaces in the filename. For example, a listing of a directory using "ls -1" gives:
ls -l *.mp3 -rw-r--r-- 1 lucky users 20168832 2006-11-03 17:54 01 Allegro.mp3 -rw-r--r-- 1 lucky users 8105984 2006-11-03 17:53 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 lucky users 10661760 2006-11-03 17:53 03 Rondo - Allegro.mp3
The following simplified script
for i in `ls -1 *.mp3` do echo $i done
The 'for' command parses every token in each line returned by `ls -l *.mp3`. Instead try a_command_that_produces_output_lines_with_spaces | while read one_line do echo $one_line done Note that the trivial example script above should produce identical output to simply entering a_command_that_produces_output_lines_with_spaces Jim
Lucky Leavell
Hi, On Monday, November 06, 2006 at 17:26:47, Lucky Leavell wrote:
The following simplified script
for i in `ls -1 *.mp3` do echo $i done
Useless use of ls. for i in *.mp3; do echo $i; done Henne -- Henne Vogelsang, http://hennevogel.de "To die. In the rain. Alone." Ernest Hemingway
On Mon, 6 Nov 2006, Henne Vogelsang wrote:
Hi,
On Monday, November 06, 2006 at 17:26:47, Lucky Leavell wrote:
The following simplified script
for i in `ls -1 *.mp3` do echo $i done
Useless use of ls.
for i in *.mp3; do echo $i; done
Thanks to all who replied; this suggestion works quite well as does Robert Lewis'. Thank you, Lucky
Lucky Leavell wrote: Hello Lucky, do you tested the conmmand " basename " ? basename --help or info basename shows some help. Best regards Wigbert
I am trying to write a simple bash script to handle files with embedded spaces in the filename. For example, a listing of a directory using "ls -1" gives:
ls -l *.mp3 -rw-r--r-- 1 lucky users 20168832 2006-11-03 17:54 01 Allegro.mp3 -rw-r--r-- 1 lucky users 8105984 2006-11-03 17:53 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 lucky users 10661760 2006-11-03 17:53 03 Rondo - Allegro.mp3
The following simplified script
for i in `ls -1 *.mp3` do echo $i done
produces the following output:
01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3
I have tried various combinations of single and double quotes to no avail including redirecting the "ls -1" listing to a file and escaping the spaces with a backslash, again to no avail.
Surely there must be some trick I can use to handle this anomoly.?
Thank you, Lucky Leavell
On 11/6/06, Lucky Leavell
I am trying to write a simple bash script to handle files with embedded spaces in the filename. For example, a listing of a directory using "ls -1" gives:
ls -l *.mp3 -rw-r--r-- 1 lucky users 20168832 2006-11-03 17:54 01 Allegro.mp3 -rw-r--r-- 1 lucky users 8105984 2006-11-03 17:53 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 lucky users 10661760 2006-11-03 17:53 03 Rondo - Allegro.mp3
The following simplified script
for i in `ls -1 *.mp3` do echo $i done
produces the following output:
01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3
I have tried various combinations of single and double quotes to no avail including redirecting the "ls -1" listing to a file and escaping the spaces with a backslash, again to no avail.
Surely there must be some trick I can use to handle this anomoly.?
Thank you, Lucky Leavell
In addition to the other responses, don't forget that find and xargs both have specific options for supporting filenames with spaces in them. Greg -- Greg Freemyer The Norcross Group Forensics for the 21st Century
Lucky Leavell wrote:
I am trying to write a simple bash script to handle files with embedded spaces in the filename. For example, a listing of a directory using "ls -1" gives:
ls -l *.mp3 -rw-r--r-- 1 lucky users 20168832 2006-11-03 17:54 01 Allegro.mp3 -rw-r--r-- 1 lucky users 8105984 2006-11-03 17:53 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 lucky users 10661760 2006-11-03 17:53 03 Rondo - Allegro.mp3
The following simplified script
for i in `ls -1 *.mp3` do echo $i done
produces the following output:
01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3
I have tried various combinations of single and double quotes to no avail including redirecting the "ls -1" listing to a file and escaping the spaces with a backslash, again to no avail.
Surely there must be some trick I can use to handle this anomoly.?
Thank you, Lucky Leavell
for FILE in '*.mp3'; do echo ${FILE}; done -- Geir A. Myrestrand
Geir, On Monday 06 November 2006 15:40, Geir A. Myrestrand wrote:
...
for FILE in '*.mp3'; do echo ${FILE}; done
You should try these things before submitting them. That will only echo (literally) *.mp3. Witness: % for FILE in '*.mp3'; do echo ${FILE}; done *.mp3 Randall Schulz
Randall R Schulz wrote:
Geir,
On Monday 06 November 2006 15:40, Geir A. Myrestrand wrote:
...
for FILE in '*.mp3'; do echo ${FILE}; done
You should try these things before submitting them. That will only echo (literally) *.mp3. Witness:
% for FILE in '*.mp3'; do echo ${FILE}; done *.mp3
Randall Schulz
I did: chiangmai:/x # env | grep SHELL SHELL=/bin/bash chiangmai:/x # touch "01 Allegro.mp3" chiangmai:/x # touch "02 Adagio un poco mosso.mp3" chiangmai:/x # touch "03 Rondo - Allegro.mp3" chiangmai:/x # ls -l total 0 -rw-r--r-- 1 root root 0 Nov 6 21:55 01 Allegro.mp3 -rw-r--r-- 1 root root 0 Nov 6 21:55 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 root root 0 Nov 6 21:56 03 Rondo - Allegro.mp3 chiangmai:/x # for FILE in '*.mp3'; do echo ${FILE}; done 01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3 chiangmai:/x # cat /etc/SuSE-release SUSE Linux Enterprise Server 10 (x86_64) VERSION = 10 However, Randall is right --on the second run it doesn't give the right output, so using for example find is the right solution here. -- Geir A. Myrestrand
On Monday 06 November 2006 19:03, Geir A. Myrestrand wrote:
Randall R Schulz wrote:
Geir,
On Monday 06 November 2006 15:40, Geir A. Myrestrand wrote:
...
for FILE in '*.mp3'; do echo ${FILE}; done
You should try these things before submitting them. That will only echo (literally) *.mp3. Witness:
% for FILE in '*.mp3'; do echo ${FILE}; done *.mp3
Randall Schulz
I did:
chiangmai:/x # env | grep SHELL SHELL=/bin/bash chiangmai:/x # touch "01 Allegro.mp3" chiangmai:/x # touch "02 Adagio un poco mosso.mp3" chiangmai:/x # touch "03 Rondo - Allegro.mp3" chiangmai:/x # ls -l total 0 -rw-r--r-- 1 root root 0 Nov 6 21:55 01 Allegro.mp3 -rw-r--r-- 1 root root 0 Nov 6 21:55 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 root root 0 Nov 6 21:56 03 Rondo - Allegro.mp3 chiangmai:/x # for FILE in '*.mp3'; do echo ${FILE}; done 01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3
Here's what's happening. You set FILE to (again, literally) *.mp3. There was one iteration of the loop (as evidenced by the fact that all the output was on one line) and since ${FILE} was not quoted, the *.mp3 was expanded as part of executing the echo command. The result I showed occurred because there were no .mp3 files in the directory where I executed it and I have the "nullglob" shell option (see the shopt command: "help shopt" or "man bash") _unset_. With this option off a glob (e.g., *.mp3) that matches no files is simply passed through to the command unchanged. If I enable nullglob, then the "echo *.mp3" produces no output (other than the newline always generated by echo). In any event, this is not a solution to the original problem of working with file names with spaces embedded. This approach works: % touch "foo bar" "bar foo" "what not" % for spaceFile in *\ *; do echo "[[$spaceFile]]"; done [[bar foo]] [[foo bar]] [[what not]] The *\ * in the for loop is just to exclude all the other files in the directory that did _not_ have spaces in their names.
...
--
Geir A. Myrestrand
Randall Schulz
Randall R Schulz wrote:
On Monday 06 November 2006 19:03, Geir A. Myrestrand wrote:
Randall R Schulz wrote:
Geir,
On Monday 06 November 2006 15:40, Geir A. Myrestrand wrote:
...
for FILE in '*.mp3'; do echo ${FILE}; done You should try these things before submitting them. That will only echo (literally) *.mp3. Witness:
% for FILE in '*.mp3'; do echo ${FILE}; done *.mp3
Randall Schulz I did:
chiangmai:/x # env | grep SHELL SHELL=/bin/bash chiangmai:/x # touch "01 Allegro.mp3" chiangmai:/x # touch "02 Adagio un poco mosso.mp3" chiangmai:/x # touch "03 Rondo - Allegro.mp3" chiangmai:/x # ls -l total 0 -rw-r--r-- 1 root root 0 Nov 6 21:55 01 Allegro.mp3 -rw-r--r-- 1 root root 0 Nov 6 21:55 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 root root 0 Nov 6 21:56 03 Rondo - Allegro.mp3 chiangmai:/x # for FILE in '*.mp3'; do echo ${FILE}; done 01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3
Here's what's happening. You set FILE to (again, literally) *.mp3. There was one iteration of the loop (as evidenced by the fact that all the output was on one line) and since ${FILE} was not quoted, the *.mp3 was expanded as part of executing the echo command.
The result I showed occurred because there were no .mp3 files in the directory where I executed it and I have the "nullglob" shell option (see the shopt command: "help shopt" or "man bash") _unset_. With this option off a glob (e.g., *.mp3) that matches no files is simply passed through to the command unchanged. If I enable nullglob, then the "echo *.mp3" produces no output (other than the newline always generated by echo).
In any event, this is not a solution to the original problem of working with file names with spaces embedded.
This approach works:
% touch "foo bar" "bar foo" "what not"
% for spaceFile in *\ *; do echo "[[$spaceFile]]"; done [[bar foo]] [[foo bar]] [[what not]]
The *\ * in the for loop is just to exclude all the other files in the directory that did _not_ have spaces in their names.
...
--
Geir A. Myrestrand
Randall Schulz
Thanks for the detailed explanation. Shell expansion is not one of my specialities. I usually use find myself when dealing with these type of issues. Two variations with find are: find . -name '*.mp3' -exec echo {} \; find . -name '*.mp3' -exec ls -l {} \; When I first used the for loop I only had one file in the directory, so I did not even notice that it would expand into just one line. -- Geir A. Myrestrand
On 2006-11-06 21:03, Geir A. Myrestrand wrote:
Randall R Schulz wrote:
Geir,
On Monday 06 November 2006 15:40, Geir A. Myrestrand wrote:
...
for FILE in '*.mp3'; do echo ${FILE}; done
You should try these things before submitting them. That will only echo (literally) *.mp3. Witness:
<snip>
However, Randall is right --on the second run it doesn't give the right output, so using for example find is the right solution here.
No, all you need to do is get rid of the quotes around *.mp3.
Darryl Gregorash wrote:
On 2006-11-06 21:03, Geir A. Myrestrand wrote:
Randall R Schulz wrote:
Geir,
On Monday 06 November 2006 15:40, Geir A. Myrestrand wrote:
...
for FILE in '*.mp3'; do echo ${FILE}; done You should try these things before submitting them. That will only echo (literally) *.mp3. Witness:
<snip>
However, Randall is right --on the second run it doesn't give the right output, so using for example find is the right solution here.
No, all you need to do is get rid of the quotes around *.mp3.
Yes, that works: # for FILE in *.mp3; do echo ${FILE}; done 01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3 -- Geir A. Myrestrand
On Monday 06 November 2006 19:19, Darryl Gregorash wrote:
On 2006-11-06 21:03, Geir A. Myrestrand wrote:
Randall R Schulz wrote:
Geir,
On Monday 06 November 2006 15:40, Geir A. Myrestrand wrote:
...
for FILE in '*.mp3'; do echo ${FILE}; done
You should try these things before submitting them. That will only echo (literally) *.mp3. Witness:
<snip>
However, Randall is right --on the second run it doesn't give the right output, so using for example find is the right solution here.
No, all you need to do is get rid of the quotes around *.mp3.
No. The original question regards files whose names have spaces in them. If you do that, you'll get the undesirable malfunctions shown in the original posting. Randall Schulz
Randall R Schulz wrote:
On Monday 06 November 2006 19:19, Darryl Gregorash wrote:
On 2006-11-06 21:03, Geir A. Myrestrand wrote:
Randall R Schulz wrote:
Geir,
On Monday 06 November 2006 15:40, Geir A. Myrestrand wrote:
...
for FILE in '*.mp3'; do echo ${FILE}; done You should try these things before submitting them. That will only echo (literally) *.mp3. Witness:
<snip> However, Randall is right --on the second run it doesn't give the right output, so using for example find is the right solution here. No, all you need to do is get rid of the quotes around *.mp3.
No. The original question regards files whose names have spaces in them. If you do that, you'll get the undesirable malfunctions shown in the original posting.
Randall Schulz
It works fine for me: # for FILE in *.mp3; do echo ${FILE}; done 01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3 -- Geir A. Myrestrand
On 2006-11-06 21:50, Randall R Schulz wrote:
On Monday 06 November 2006 19:19, Darryl Gregorash wrote:
No, all you need to do is get rid of the quotes around *.mp3.
No. The original question regards files whose names have spaces in them. If you do that, you'll get the undesirable malfunctions shown in the original posting.
Mileage must vary, then: raven@static24-89-67-198:~> ./thingy Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u Georg Friederich Händel-Water Music Suite.m3u Johannes Brahms-Symphony no 1 in C minor Op 68 Thomas Sanderling Philharmonia Orchestra.m3u Johannes Brahms-Symphony no 2 in D major Op 73 Thomas Sanderling Philharmonic Orchestra.m3u Johannes Brahms-Symphony no 3 in F major Op 90 Tragic Overture Op 81 Thomas Sanderling Philharmonia Orchestra.m3u Johannes Brahms-Symphony no 4 in E minor Op 98 Thomas Sanderling Philharmonia Orchestra.m3u Mikhail IppolitovIvanov-Liturgy of St John Chrysostom Op 37 Vespers Op 43.m3u St Petersburg Chamber Choir-Russian Easter.m3u The Mamas The Papas-Beautiful Thing.m3u The Mamas The Papas-California Dreamin.m3u The Mamas The Papas-If You Can Believe Your Eyes and Ears.m3u The Tallis Scholars-Allegri Miserere Palestrina Missa Papae Marcelli Mundy Vox Patris Caelestis.m3u Virtuosi di Praga-Music for the Kings of Spain.m3u Westminster Abbey Choir and the Abbey Consort-Miserere.m3u Wolfgang Amadeus Mozart-Requiem Slovak Philharmonic Orchestra.m3u raven@static24-89-67-198:~> cat thingy cd /pub/music/mp3 for i in *.m3u; do echo $i; done cd ~
On Monday 06 November 2006 20:04, Darryl Gregorash wrote:
On 2006-11-06 21:50, Randall R Schulz wrote:
On Monday 06 November 2006 19:19, Darryl Gregorash wrote:
No, all you need to do is get rid of the quotes around *.mp3.
No. The original question regards files whose names have spaces in them. If you do that, you'll get the undesirable malfunctions shown in the original posting.
Mileage must vary, then:
Not really.
...
raven@static24-89-67-198:~> cat thingy cd /pub/music/mp3 for i in *.m3u; do echo $i; done cd ~
Do something other than echo in the loopl. Try applying sum, or ls, e.g. You'll find unless you quote $i it won't work. Randall Schulz
Randall R Schulz wrote:
On Monday 06 November 2006 20:04, Darryl Gregorash wrote:
On 2006-11-06 21:50, Randall R Schulz wrote:
No, all you need to do is get rid of the quotes around *.mp3. No. The original question regards files whose names have spaces in
On Monday 06 November 2006 19:19, Darryl Gregorash wrote: them. If you do that, you'll get the undesirable malfunctions shown in the original posting. Mileage must vary, then:
Not really.
...
raven@static24-89-67-198:~> cat thingy cd /pub/music/mp3 for i in *.m3u; do echo $i; done cd ~
Do something other than echo in the loopl. Try applying sum, or ls, e.g. You'll find unless you quote $i it won't work.
Randall Schulz
Randall is right, you need to quote the variable if you use it as an argument to a program that expects one token. An example is if you want to copy the MP3 files, the first argument to cp must be the full filename, not just the first token of the filename... However, as long as echo is sufficient then it will work as pointed out by Randall. -- Geir A. Myrestrand
...
raven@static24-89-67-198:~> cat thingy cd /pub/music/mp3 for i in *.m3u; do echo $i; done cd ~
Do something other than echo in the loopl. Try applying sum, or ls, e.g. You'll find unless you quote $i it won't work.
Randall Schulz
OK, here is my complete scipt which runs under bash on an Ubuntu 6.06 system where I have the mp32ogg utility to convert mp3 to ogg files which k3b can handle: for i in *.mp3 do echo $i mp32ogg $i done whch works fine if the only craziness is embedded spaces in the file name; it failed when there were parentheses but, since that is rare in my situation, I can live with it. (Of course, if the single quoted $i would work there ... I'm off to try it!) Thank you, Lucky
On Tue, 7 Nov 2006, Lucky Leavell wrote:
...
raven@static24-89-67-198:~> cat thingy cd /pub/music/mp3 for i in *.m3u; do echo $i; done cd ~
Do something other than echo in the loopl. Try applying sum, or ls, e.g. You'll find unless you quote $i it won't work.
Randall Schulz
OK, here is my complete scipt which runs under bash on an Ubuntu 6.06 system where I have the mp32ogg utility to convert mp3 to ogg files which k3b can handle:
for i in *.mp3 do echo $i mp32ogg $i done
whch works fine if the only craziness is embedded spaces in the file name; it failed when there were parentheses but, since that is rare in my situation, I can live with it. (Of course, if the single quoted $i would work there ... I'm off to try it!)
Thank you, Lucky
I tried it and the quotes make no difference but, again, that is not a major problem for this particular application. Thank you, Lucky
On Monday 06 November 2006 21:11, Lucky Leavell wrote:
...
OK, here is my complete scipt which runs under bash on an Ubuntu 6.06 system where I have the mp32ogg utility to convert mp3 to ogg files which k3b can handle:
for i in *.mp3 do echo $i mp32ogg $i done
whch works fine if the only craziness is embedded spaces in the file name; it failed when there were parentheses but, since that is rare in my situation, I can live with it. (Of course, if the single quoted $i would work there ... I'm off to try it!)
Just make it a habit to quote variables. Then you won't have to worry about any characters that appear in the variables. Remember, there is only one character code forbidden in Unix / Linux file names: the slash. % touch "foo bar" "bar foo" "what not" No quotes on variable reference: % for spaceFile in *\ *; do ll $spaceFile; done ls: bar: No such file or directory ls: foo: No such file or directory ls: foo: No such file or directory ls: bar: No such file or directory ls: what: No such file or directory ls: not: No such file or directory With quoted variable reference: % for spaceFile in *\ *; do ll "$spaceFile"; done -rw-r--r-- 1 rschulz users 0 2006-11-06 19:12 bar foo -rw-r--r-- 1 rschulz users 0 2006-11-06 19:12 foo bar -rw-r--r-- 1 rschulz users 0 2006-11-06 19:12 what not If the code fragment you gave above works, then what appears to be spaces are some other character code, not an ASCII space (040 / 0x20). There are other character codes that make no mark and are not zero-width. Perhaps your file names use one of them. To find out, pipe the names into "od -ab" (use a fixed-width font to view this and note that the numeric codes are octal, not decimal or hex): Normal spaces (octal 020): % ls *\ * |od -ab 0000000 b a r sp f o o nl f o o sp b a r nl 142 141 162 040 146 157 157 012 146 157 157 040 142 141 162 012 0000020 w h a t sp n o t nl 167 150 141 164 040 156 157 164 012 With en-spaces (a space whose width in proportionally spaced fonts is equal to that of the lower-case 'n') in the file name 'en space': % ls -lt |head -5 total 5296 -rw-r--r-- 1 rschulz users 0 2006-11-06 21:59 en space -rw-r--r-- 1 rschulz users 0 2006-11-06 19:12 bar foo -rw-r--r-- 1 rschulz users 0 2006-11-06 19:12 what not -rw-r--r-- 1 rschulz users 0 2006-11-06 19:12 foo bar % echo en space |od -ab 0000000 e n b nul stx s p a c e nl 145 156 342 200 202 163 160 141 143 145 012 Now see how the absence of quote marks does not cause the file named "en space" to be interpreted as two arguments: % for spaceFile in en*; do ll $spaceFile; done -rw-r--r-- 1 rschulz users 0 2006-11-06 21:59 en space Character set issues always complicate life... Lesson: ALWAYS QUOTE VARIABLE REFERENCES WHOSE CONTENT YOU DO NOT FULLY CONTROL. (Sorry for shouting. Sometimes it's called for...)
Thank you, Lucky
Randall Schulz
On 2006-11-07 00:06, Randall R Schulz wrote:
On Monday 06 November 2006 21:11, Lucky Leavell wrote:
...
OK, here is my complete scipt which runs under bash on an Ubuntu 6.06 system where I have the mp32ogg utility to convert mp3 to ogg files which k3b can handle:
for i in *.mp3 do echo $i mp32ogg $i done
whch works fine if the only craziness is embedded spaces in the file name; it failed when there were parentheses but, since that is rare in my situation, I can live with it. (Of course, if the single quoted $i would work there ... I'm off to try it!)
<snip>
If the code fragment you gave above works, then what appears to be spaces are some other character code, not an ASCII space (040 / 0x20). There are other character codes that make no mark and are not zero-width. Perhaps your file names use one of them.
That simply is not possible. The list is determined by "for i in <list>" and that list is determined prior to the actual parsing of any commands inside the for loop. If it is a literal space for the ls command, then it must be a literal space for any other command. But the variation of results reported here, and your comments in particular, have made me wonder what effect the quotes will have placed in different places inside the script. First I changed my test script to execute a "ls -l $i" after the echo, and came up with this sort of result: Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u ls: Antonio: No such file or directory ls: Vivaldi-Gloria: No such file or directory ls: Dixit: No such file or directory ls: Dominus: No such file or directory (etc) exactly as you stated. The best I can think of is that the ls command takes a list, ie. the actual ls commandline format is not "ls [options] [filespec]", rather it is "ls [options] [list_of_filespecs]". Any command behaving in the same way, such as cp, will fail in the same way. When $i in the ls command is placed inside double quotes, I get the expected ls output: ~>cat thingy cd /pub/music/mp3 for i in *.m3u; do echo $i; ls -l "$i"; done cd ~ ~>./thingy Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u -rw-r--r-- 1 raven users 5364 2005-11-05 01:31 Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u (etc) If the double quotes are replaced with single quotes, each ls command returns an error, "ls: $i: No such file or directory". I've forgotten the subtle nuances of quotes in the bash commandline to remember why this happens. When the for loop is written the following way (and these can be single or double quotes, it does not matter), then there is a single variable in the list of the for loop, so the echo and ls commands are actually run just once each: for i in '*.m3u'; do echo $i; ls -l $i; done and the output becomes: ~> ./thingy Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u Georg Friederich Händel-Water Music Suite.m3u Johannes Brahms-Symphony no 1 in C minor Op 68 Thomas Sanderling Philharmonia Orchestra.m3u Johannes Brahms-Symphony no 2 in D major Op 73 Thomas Sanderling Philharmonic Orchestra.m3u Johannes Brahms-Symphony no 3 in F major Op 90 Tragic Overture Op 81 Thomas Sanderling Philharmonia Orchestra.m3u Johannes Brahms-Symphony no 4 in E minor Op 98 Thomas Sanderling Philharmonia Orchestra.m3u Mikhail IppolitovIvanov-Liturgy of St John Chrysostom Op 37 Vespers Op 43.m3u St Petersburg Chamber Choir-Russian Easter.m3u The Mamas The Papas-Beautiful Thing.m3u The Mamas The Papas-California Dreamin.m3u The Mamas The Papas-If You Can Believe Your Eyes and Ears.m3u The Tallis Scholars-Allegri Miserere Palestrina Missa Papae Marcelli Mundy Vox Patris Caelestis.m3u Virtuosi di Praga-Music for the Kings of Spain.m3u Westminster Abbey Choir and the Abbey Consort-Miserere.m3u Wolfgang Amadeus Mozart-Requiem Slovak Philharmonic Orchestra.m3u -rw-r--r-- 1 raven users 5364 2005-11-05 01:31 Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u -rw-r--r-- 1 raven users 624 2005-11-19 03:12 Georg Friederich Händel-Water Music Suite.m3u (etc)
Lesson: ALWAYS QUOTE VARIABLE REFERENCES WHOSE CONTENT YOU DO NOT FULLY CONTROL.
So, that is absolutely correct, but with the qualification that what you get does depend on where and how you do the quoting, and AFAICT there is no hard and fast rule that gives a single correct way to use quotes. Did someone say the bash commandline isn't interesting and fun? :-)
On Monday 06 November 2006 23:55, Darryl Gregorash wrote:
...
If the code fragment you gave above works, then what appears to be spaces are some other character code, not an ASCII space (040 / 0x20). There are other character codes that make no mark and are not zero-width. Perhaps your file names use one of them.
That simply is not possible. The list is determined by "for i in <list>" and that list is determined prior to the actual parsing of any commands inside the for loop. If it is a literal space for the ls command, then it must be a literal space for any other command. But the variation of results reported here, and your comments in particular, have made me wonder what effect the quotes will have placed in different places inside the script.
In this fragment: % for VAR_NAME in GLOB ... The glob is expanded by the for loop processing and is not subject to further argument list parsing for the purpose of producing successive values to assign to VAR_NAME, so spaces in the names don't confuse the iteration. There is always exactly one iteration for each file name matched by GLOB. Nonetheless, my point was that if you have a character such as an en-space (or it's fatter sibling, em-space, which is the width of a lower-case 'm'), it can be expanded unquoted in an argument list without the en-space causing a break in the argument as a regular space would.
First I changed my test script to execute a "ls -l $i" after the echo, and came up with this sort of result:
Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u ls: Antonio: No such file or directory ls: Vivaldi-Gloria: No such file or directory ls: Dixit: No such file or directory ls: Dominus: No such file or directory (etc)
exactly as you stated.
The best I can think of is that the ls command takes a list, ie. the actual ls commandline format is not "ls [options] [filespec]", rather it is "ls [options] [list_of_filespecs]". Any command behaving in the same way, such as cp, will fail in the same way. When $i in the ls command is placed inside double quotes, I get the expected ls output:
Absolutely not. Ls is not special in this way. The only special treatment is the expansion of the glob pattern and the assignment of the resulting file names to the for-loop variable without further argument parsing occurring. Once the for-loop variable is referenced in the body of the loop, quoting again becomes significant and an unqutoed reference when the value contains shell-significant syntactic characters (space, [, *, *, ? | and so on) will yield to unexpected and undesired results.
~>cat thingy cd /pub/music/mp3 for i in *.m3u; do echo $i; ls -l "$i"; done cd ~ ~>./thingy Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u -rw-r--r-- 1 raven users 5364 2005-11-05 01:31 Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u (etc)
If the double quotes are replaced with single quotes, each ls command returns an error, "ls: $i: No such file or directory". I've forgotten the subtle nuances of quotes in the bash commandline to remember why this happens.
Within double-quotes variable and command-to-argument expansion (back-ticks or $( ... ) ) are expanded. Within single quotes _nothing_ is expanded.
When the for loop is written the following way (and these can be single or double quotes, it does not matter), then there is a single variable in the list of the for loop, so the echo and ls commands are actually run just once each:
for i in '*.m3u'; do echo $i; ls -l $i; done
and the output becomes:
~> ./thingy Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u Georg ... -rw-r--r-- 1 raven users 5364 2005-11-05 01:31 Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u -rw-r--r-- 1 raven users 624 2005-11-19 03:12 Georg Friederich Händel-Water Music Suite.m3u (etc)
In this case, the glob expansion of *.m3u is happening upon each unquoted reference in the body of the loop.
Lesson: ALWAYS QUOTE VARIABLE REFERENCES WHOSE CONTENT YOU DO NOT FULLY CONTROL.
So, that is absolutely correct, but with the qualification that what you get does depend on where and how you do the quoting, and AFAICT there is no hard and fast rule that gives a single correct way to use quotes.
No, there's not. One cannot write non-trivial programs in any programming language blindly, based only on rules-of-thumb without really knowing that language and whatever subtleties it may have. BASH has a few...
Did someone say the bash commandline isn't interesting and fun? :-)
I think so. I do tons of BASH programming. Randall Schulz
On 2006-11-07 09:46, Randall R Schulz wrote: OK, you've answered a lot of questions, but raised a few more. (No offence intended, but I do think Joachim has given a better explanation of this.)
<snip> Nonetheless, my point was that if you have a character such as an en-space (or it's fatter sibling, em-space, which is the width of a lower-case 'm'), it can be expanded unquoted in an argument list without the en-space causing a break in the argument as a regular space would.
Absolutely not. Ls is not special in this way. The only special treatment is the expansion of the glob pattern... I didn't say ls is special in any way; what I did say (not very well, I admit) is that the ls commandline syntax is "ls ...
In this instance, we are talking about filenames that clearly have regular spaces as part of their filenames, so I don't see how speaking of en- or em-spaces is relevant. I don't believe there is any way a regular space could be translated to any other kind of space just as part of any glob processing. list of filespecs>" and thus, without any (double) quotes around the variable name, "ls ... $i" will produce unexpected results if the value of $i is a filespec containing spaces.
<snip>
for i in '*.m3u'; do echo $i; ls -l $i; done
In this case, the glob expansion of *.m3u is happening upon each unquoted reference in the body of the loop.
As I just found running "echo *.m3u" in that directory, on the commandline. -That- is something I certainly did not expect -- :-)
Did someone say the bash commandline isn't interesting and fun? :-)
I think so. I do tons of BASH programming. Fortunately for you, I do not do all that much, or you might suddenly find your inbox full of messages starting "Help, I cannot figure out what this thing is doing...." :-)
On Tuesday 07 November 2006 09:58, Darryl Gregorash wrote:
On 2006-11-07 09:46, Randall R Schulz wrote:
...
<snip> Nonetheless, my point was that if you have a character such as an en-space (or it's fatter sibling, em-space, which is the width of a lower-case 'm'), it can be expanded unquoted in an argument list without the en-space causing a break in the argument as a regular space would.
In this instance, we are talking about filenames that clearly have regular spaces as part of their filenames, so I don't see how speaking of en- or em-spaces is relevant. I don't believe there is any way a regular space could be translated to any other kind of space just as part of any glob processing.
Non-ASCII space characters are the only hypothesis that explains this: On Monday 06 November 2006 21:11, Lucky Leavell wrote:
for i in *.mp3 do echo $i mp32ogg $i done
whch works fine if the only craziness is embedded spaces in the file name; it failed when there were parentheses but, since that is rare in my situation, I can live with it. (Of course, if the single quoted $i would work there ... I'm off to try it!)
If that "works fine" with "embedded spaces," then those spaces must not be ASCII spaces (040 / 0x20) but instead something like an en- or em-space. Furthermore, of all the contexts in which non-ASCII spaces might show up, coming from a media source such as a CD or music download seems more plausible than someone typing them at a terminal or even a GUI application (I had to open an HTML editor, enter the HTML entity names, switch to the WYSIWYG mode and copy the characters to get them, since I don't know all the extended compose codes for inputting them via the keyboard).
...
Did someone say the bash commandline isn't interesting and fun? :-)
I think so. I do tons of BASH programming.
Fortunately for you, I do not do all that much, or you might suddenly find your inbox full of messages starting "Help, I cannot figure out what this thing is doing...." :-)
I'm happy to answer questions, but please, post them here. Randall Schulz
On Monday 06 November 2006 21:55, Darryl Gregorash wrote:
On 2006-11-07 00:06, Randall R Schulz wrote:
On Monday 06 November 2006 21:11, Lucky Leavell wrote:
...
OK, here is my complete scipt which runs under bash on an Ubuntu 6.06 system where I have the mp32ogg utility to convert mp3 to ogg files which k3b can handle:
for i in *.mp3 do echo $i mp32ogg $i done
whch works fine if the only craziness is embedded spaces in the file name; it failed when there were parentheses but, since that is rare in my situation, I can live with it. (Of course, if the single quoted $i would work there ... I'm off to try it!)
<snip>
If the code fragment you gave above works, then what appears to be spaces are some other character code, not an ASCII space (040 / 0x20). There are other character codes that make no mark and are not zero-width. Perhaps your file names use one of them.
That simply is not possible. The list is determined by "for i in <list>" and that list is determined prior to the actual parsing of any commands inside the for loop. If it is a literal space for the ls command, then it must be a literal space for any other command. But the variation of results reported here, and your comments in particular, have made me wonder what effect the quotes will have placed in different places inside the script.
First I changed my test script to execute a "ls -l $i" after the echo, and came up with this sort of result:
Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u ls: Antonio: No such file or directory ls: Vivaldi-Gloria: No such file or directory ls: Dixit: No such file or directory ls: Dominus: No such file or directory (etc)
exactly as you stated.
The best I can think of is that the ls command takes a list, ie. the actual ls commandline format is not "ls [options] [filespec]", rather it is "ls [options] [list_of_filespecs]". Any command behaving in the same way, such as cp, will fail in the same way. When $i in the ls command is placed inside double quotes, I get the expected ls output:
~>cat thingy cd /pub/music/mp3 for i in *.m3u; do echo $i; ls -l "$i"; done cd ~ ~>./thingy Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u -rw-r--r-- 1 raven users 5364 2005-11-05 01:31 Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u (etc)
If the double quotes are replaced with single quotes, each ls command returns an error, "ls: $i: No such file or directory". I've forgotten the subtle nuances of quotes in the bash commandline to remember why this happens.
When the for loop is written the following way (and these can be single or double quotes, it does not matter), then there is a single variable in the list of the for loop, so the echo and ls commands are actually run just once each:
for i in '*.m3u'; do echo $i; ls -l $i; done
and the output becomes:
~> ./thingy Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u Georg Friederich Händel-Water Music Suite.m3u Johannes Brahms-Symphony no 1 in C minor Op 68 Thomas Sanderling Philharmonia Orchestra.m3u Johannes Brahms-Symphony no 2 in D major Op 73 Thomas Sanderling Philharmonic Orchestra.m3u Johannes Brahms-Symphony no 3 in F major Op 90 Tragic Overture Op 81 Thomas Sanderling Philharmonia Orchestra.m3u Johannes Brahms-Symphony no 4 in E minor Op 98 Thomas Sanderling Philharmonia Orchestra.m3u Mikhail IppolitovIvanov-Liturgy of St John Chrysostom Op 37 Vespers Op 43.m3u St Petersburg Chamber Choir-Russian Easter.m3u The Mamas The Papas-Beautiful Thing.m3u The Mamas The Papas-California Dreamin.m3u The Mamas The Papas-If You Can Believe Your Eyes and Ears.m3u The Tallis Scholars-Allegri Miserere Palestrina Missa Papae Marcelli Mundy Vox Patris Caelestis.m3u Virtuosi di Praga-Music for the Kings of Spain.m3u Westminster Abbey Choir and the Abbey Consort-Miserere.m3u Wolfgang Amadeus Mozart-Requiem Slovak Philharmonic Orchestra.m3u -rw-r--r-- 1 raven users 5364 2005-11-05 01:31 Antonio Vivaldi-Gloria Dixit Dominus Magnificat Choir of Kings College Cambridge and the Academy of Ancient Music.m3u -rw-r--r-- 1 raven users 624 2005-11-19 03:12 Georg Friederich Händel-Water Music Suite.m3u (etc)
Lesson: ALWAYS QUOTE VARIABLE REFERENCES WHOSE CONTENT YOU DO NOT FULLY CONTROL.
So, that is absolutely correct, but with the qualification that what you get does depend on where and how you do the quoting, and AFAICT there is no hard and fast rule that gives a single correct way to use quotes.
Did someone say the bash commandline isn't interesting and fun? :-)
This is a very interesting thread. Parts are going into my "Good Stuff' sub folder. When in doubt-quote;thoughtfully. A relative constant, how nice. Thanks, Jerome
Lucky Leavell wrote:
OK, here is my complete scipt which runs under bash on an Ubuntu 6.06 system where I have the mp32ogg utility to convert mp3 to ogg files which k3b can handle:
for i in *.mp3 do echo $i mp32ogg $i done
whch works fine if the only craziness is embedded spaces in the file name; it failed when there were parentheses but, since that is rare in my situation, I can live with it. (Of course, if the single quoted $i would work there ... I'm off to try it!)
You need to use double quotes: In double quotes, shell variables are expanded, i.e., their value is inserted. In single quotes, variables are not expanded. The correct version of your script is: for i in *.mp3 do echo "$i" mp32ogg "$i" done Some might argue that the double quotes at echo are superfluous; but they are not: They are proper shell programming. If there is a file with two spaces in it, it is now output the proper way; otherwise the two spaces would have been turned into one space. The lesson: If you have a shall variable and pass it as an argument to a command, ALWAYS include it in double quotes UNLESS if you want the value broken along its white space. Everything else is sloppy programming and might bite you some time in the future. You can do that in interactive usage, but refrain from it in shell scripts. Oh yes, and within quotes, shell wildcards are not expanded. Therefore you must not put any quotes around *.mp3, as some posters did it. Joachim (who wrote his first shell script in the mid-80s :-) ----- PS: And something fitting from my quote collection. [You can find the reference on Google Groups. :-)] From: Larry Wall Drew Mills writes: : A contest to see who could write the most useful script that could : actually be used in the most languages *as is* [...] I've written some scripts that work in 582 different languages, all of them named sh. -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Joachim Schrod Email: jschrod@acm.org Roedermark, Germany
Lucky Leavell wrote:
I am trying to write a simple bash script to handle files with embedded spaces in the filename. For example, a listing of a directory using "ls -1" gives:
ls -l *.mp3 -rw-r--r-- 1 lucky users 20168832 2006-11-03 17:54 01 Allegro.mp3 -rw-r--r-- 1 lucky users 8105984 2006-11-03 17:53 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 lucky users 10661760 2006-11-03 17:53 03 Rondo - Allegro.mp3
The following simplified script
for i in `ls -1 *.mp3` do echo $i done
lose the 'ls': for i in *.mp3;do echo $i done Or, if you're paranoid: for i in *.mp3;do echo "$i" done -- Until later, Geoffrey Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. - Benjamin Franklin
Il giorno Mon, 6 Nov 2006 17:26:47 -0500 (EST)
Lucky Leavell
I am trying to write a simple bash script to handle files with embedded spaces in the filename. For example, a listing of a directory using "ls -1" gives:
ls -l *.mp3 -rw-r--r-- 1 lucky users 20168832 2006-11-03 17:54 01 Allegro.mp3 -rw-r--r-- 1 lucky users 8105984 2006-11-03 17:53 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 lucky users 10661760 2006-11-03 17:53 03 Rondo - Allegro.mp3
The following simplified script
for i in `ls -1 *.mp3` do echo $i done
produces the following output:
01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3
I have tried various combinations of single and double quotes to no avail including redirecting the "ls -1" listing to a file and escaping the spaces with a backslash, again to no avail.
Surely there must be some trick I can use to handle this anomoly.?
Read Advanced Bash-scripting guide by Mendel Cooper!
Thank you, Lucky Leavell
emilio
On Tuesday 07 November 2006 05:44, em.conti@tin.it wrote:
...
Surely there must be some trick I can use to handle this anomoly.?
Read Advanced Bash-scripting guide by Mendel Cooper!
I like "Linux Shell Scripting with Bash" by Ken O. Burtch, ISBN 0-672-32642-6, Developer's Library.
emilio
Randall Schulz
On Mon, 2006-11-06 at 17:26 -0500, Lucky Leavell wrote:
I am trying to write a simple bash script to handle files with embedded spaces in the filename. For example, a listing of a directory using "ls -1" gives:
ls -l *.mp3 -rw-r--r-- 1 lucky users 20168832 2006-11-03 17:54 01 Allegro.mp3 -rw-r--r-- 1 lucky users 8105984 2006-11-03 17:53 02 Adagio un poco mosso.mp3 -rw-r--r-- 1 lucky users 10661760 2006-11-03 17:53 03 Rondo - Allegro.mp3
The following simplified script
for i in `ls -1 *.mp3` do echo $i done
produces the following output:
01 Allegro.mp3 02 Adagio un poco mosso.mp3 03 Rondo - Allegro.mp3
Replace them, do your work and either move them back or leave them. Personally I find it a problem with burning cdroms for my Sony mp3 player. # Loop over all files in a directory for i in * ; do # $i = current filename LOWERNAME=`echo -n $i| tr ' ' '_'` # mv = rename file mv $i $LOWERNAME done
-- ___ _ _ _ ____ _ _ _ | | | | [__ | | | |___ |_|_| ___] | \/
participants (15)
-
Carl William Spitzer IV -
Darryl Gregorash -
em.conti＠tin.it -
Geir A. Myrestrand -
Geoffrey -
Greg Freemyer -
Henne Vogelsang -
Jim Cunning -
Joachim Schrod -
Lucky Leavell -
Patrick Shanahan -
Randall R Schulz -
Robert Lewis -
Susemail -
Wigbert Lindenbauer