[opensuse] unexplained tar Error
Hi everyone, Today I thought I would take on the long-outstanding (and I though simple) task of fixing my backup script to allow file names with spaces. I tried running tar with "" around the file names manually and it worked! So I though "Oh I can simply add \" to either side of the filenames in my script" where \ is escaping the " so the script does not treat it as a string. Which I did and it changed the command from this: tar --exclude /home/user2/filetoexclude.txt -V tape-label-here -cvf /dev/st2 /srv/backups/backup-script.sh /home/user/Folder with spaces/ /home/user2/ to this: tar --exclude "/home/user2/filetoexclude.txt" -V tape-label-here -cvf /dev/st2 "/srv/backups/backup-script.sh" "/home/user/Folder with spaces/" "/home/user2/" Which I then run by pasting it in the terminal it works fine. "Great this is easy" I thought. but NOOO! said tar and bash (Secretly laughing at me)!!! This is the problem: When I run the command in a bash script (this is the code that generated the above command): tar $ex_list -V $label -cvf $tape_dev $file_list This is what I get when running the script: server5:/ # bash /srv/backups/backup-script.sh tar --exclude "/home/user2/filetoexclude.txt" -V 2016-12-08-14-49-home -cvf /dev/st2 "/srv/backups/backup-script.sh" "/home/user/Folder with spaces/" "/home/user2/" tar: "/srv/backups/backup-script.sh": Cannot stat: No such file or directory tar: "/home/user/Folder with spaces/": Cannot stat: No such file or directory tar: "/home/user2/": Cannot stat: No such file or directory tar: Exiting with failure status due to previous errors If I remove the quotes, tar works as it did previously but then I cannot backup anything with a space in the name. But the weird thing is as I said previously, if I copy paste it manually in the terminal it works fine!! Ack! Keep in mind that all of this is being run as root and in the root of the file system. I also tried with single quotes ' instead of " and this made no difference and has the exact same problem. Does anyone have any clue why this happens and how I may troubleshoot it? Thanks Paul -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Dec 8, 2016 at 6:07 PM, Paul Groves <paul.groves.787@gmail.com> wrote:
Hi everyone,
Today I thought I would take on the long-outstanding (and I though simple) task of fixing my backup script to allow file names with spaces.
Welcome to real world of wonderful Unix :)
Does anyone have any clue why this happens and how I may troubleshoot it?
Use --files-from=file to read file names from file and if file names may include newline, use --null additionally (and create file accordingly). See GNU tar documentation for details. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 08/12/16 15:25, Andrei Borzenkov wrote:
On Thu, Dec 8, 2016 at 6:07 PM, Paul Groves <paul.groves.787@gmail.com> wrote:
Hi everyone,
Today I thought I would take on the long-outstanding (and I though simple) task of fixing my backup script to allow file names with spaces. Welcome to real world of wonderful Unix :)
Does anyone have any clue why this happens and how I may troubleshoot it?
Use --files-from=file to read file names from file and if file names may include newline, use --null additionally (and create file accordingly). See GNU tar documentation for details.
Hi Andrei, Unfortunately not an easy solution for me as I am pulling the file lists from a database. The way the backups work on this system is that the backup job is executed by cron This then gets the job details from SQL (which tape drive, files to backup, excluded files etc). The list of files and the excluded file lists are saved to two separate arrays Each array is then converted to a list separated by spaces (e.g. "file1" "file2" "folder1" etc). Although, I already tried using two text files --files-from=files.txt and --exclude-from=exclude.txt and this worked fine. Perhaps this is a problem with the way bash handles quotes "" ? Paul -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/08/2016 10:38 AM, Paul Groves wrote:
Perhaps this is a problem with the way bash handles quotes "" ?
Indeed, or a way that you perceive the use of quotes and delimited quotes and nested quotes as opposed to the (deterministic and documented but often non-intuitive) way BASH perceives the use of quotes. That has bitten many generations of back programmers. It gets worse of you use functions! -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
08.12.2016 18:38, Paul Groves пишет:
On 08/12/16 15:25, Andrei Borzenkov wrote:
On Thu, Dec 8, 2016 at 6:07 PM, Paul Groves <paul.groves.787@gmail.com> wrote:
Hi everyone,
Today I thought I would take on the long-outstanding (and I though simple) task of fixing my backup script to allow file names with spaces. Welcome to real world of wonderful Unix :)
Does anyone have any clue why this happens and how I may troubleshoot it?
Use --files-from=file to read file names from file and if file names may include newline, use --null additionally (and create file accordingly). See GNU tar documentation for details.
Hi Andrei,
Unfortunately not an easy solution for me as I am pulling the file lists from a database.
The way the backups work on this system is that the backup job is executed by cron This then gets the job details from SQL (which tape drive, files to backup, excluded files etc). The list of files and the excluded file lists are saved to two separate arrays
So put all these strings in temporary file instead of array; where is the problem?
Each array is then converted to a list separated by spaces (e.g. "file1" "file2" "folder1" etc).
Although, I already tried using two text files --files-from=files.txt and --exclude-from=exclude.txt and this worked fine.
Perhaps this is a problem with the way bash handles quotes "" ?
Yes. Traditional shell is the worst language to work with data containing spaces. You may consider using zsh which is compatible but has much more sane quoting rules. If you are stuck with bash, use arrays foo=("a b c" "d e f") bor@bor-Latitude-E5450:~$ LC_ALL=C tar cf /tmp/foo "${foo[@]}" tar: a b c: Cannot stat: No such file or directory tar: d e f: Cannot stat: No such file or directory -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/08/2016 10:07 AM, Paul Groves wrote:
Does anyone have any clue why this happens and how I may troubleshoot it?
More than 25 years ago, the Unix Systems Group at bell stopped using TAR in favour of CPIO. I'm not sure how they handled the tape interface. Problably the user 'rmt'. I suspect they were doing disk-to-disk-to tape snapshots as this was before we had things like LVM or CoW snapshotting. How they actually Imaged to tape i'm not sure. But the distributions I saw from them, from then on, were CPIO on the tape, not TAR. Yes 'find' + 'cpio' can be very flexible! Check the man page for 'rmt'. If you don't have an obsession about TAR, this may be an alternative. How ever if your hurdle is the use of '-null' to deal with whole filename strings and delimiting them then you have a deeper problem with your scripting and you use of quotes and escaped quotes. Yes, we still see 'tarballs' as a means of delivering packages. An unfortunate fact of life ... -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 08/12/16 16:23, Anton Aylward wrote:
On 12/08/2016 10:07 AM, Paul Groves wrote:
Does anyone have any clue why this happens and how I may troubleshoot it?
More than 25 years ago, the Unix Systems Group at bell stopped using TAR in favour of CPIO. I'm not sure how they handled the tape interface. Problably the user 'rmt'. I suspect they were doing disk-to-disk-to tape snapshots as this was before we had things like LVM or CoW snapshotting. How they actually Imaged to tape i'm not sure. But the distributions I saw from them, from then on, were CPIO on the tape, not TAR.
Yes 'find' + 'cpio' can be very flexible!
Check the man page for 'rmt'.
If you don't have an obsession about TAR, this may be an alternative.
How ever if your hurdle is the use of '-null' to deal with whole filename strings and delimiting them then you have a deeper problem with your scripting and you use of quotes and escaped quotes.
Yes, we still see 'tarballs' as a means of delivering packages. An unfortunate fact of life ...
Success!!!! Solution found. http://stackoverflow.com/questions/2355148/run-a-string-as-a-command-within-... All I had to do was change this line in my script: tar $ex_list -V $label -cvf $tape_dev $file_list to this: eval "tar $ex_list -V $label -cvf $tape_dev $file_list" Now tar works fine! yay :) I wil run a test backup and post back to confirm the whole backup process was successful. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 08/12/16 17:30, Paul Groves wrote:
On 08/12/16 16:23, Anton Aylward wrote:
On 12/08/2016 10:07 AM, Paul Groves wrote:
Does anyone have any clue why this happens and how I may troubleshoot it?
More than 25 years ago, the Unix Systems Group at bell stopped using TAR in favour of CPIO. I'm not sure how they handled the tape interface. Problably the user 'rmt'. I suspect they were doing disk-to-disk-to tape snapshots as this was before we had things like LVM or CoW snapshotting. How they actually Imaged to tape i'm not sure. But the distributions I saw from them, from then on, were CPIO on the tape, not TAR.
Yes 'find' + 'cpio' can be very flexible!
Check the man page for 'rmt'.
If you don't have an obsession about TAR, this may be an alternative.
How ever if your hurdle is the use of '-null' to deal with whole filename strings and delimiting them then you have a deeper problem with your scripting and you use of quotes and escaped quotes.
Yes, we still see 'tarballs' as a means of delivering packages. An unfortunate fact of life ...
Success!!!! Solution found.
http://stackoverflow.com/questions/2355148/run-a-string-as-a-command-within-...
All I had to do was change this line in my script: tar $ex_list -V $label -cvf $tape_dev $file_list
to this: eval "tar $ex_list -V $label -cvf $tape_dev $file_list"
Now tar works fine! yay :)
I wil run a test backup and post back to confirm the whole backup process was successful.
Alas! No luck. Files with spaces in their names work but not directories! (even though the output in the log shows the identical command which works when typed in manually). It's time to have a beer now I think. Andreas, I will give your suggestion using the command touch a go tomorrow. Thanks Paul -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Dec 8, 2016 at 11:23 AM, Anton Aylward <opensuse@antonaylward.com> wrote:
Yes 'find' + 'cpio' can be very flexible!
Check the man page for 'rmt'.
If you're going to/from tape, add buffer or mbuffer to the toolbox. It sits between cpio and the tape drive and helps ensure data is fed to the tape in appropriate size chunks. Of the 2, I like mbuffer: http://www.maier-komor.de/mbuffer.html The last source tarball is from just last month, so it is still maintained: http://www.maier-komor.de/software/mbuffer/mbuffer-20161115.tgz And it is part of openSUSE for at least the last 7 or 8 years. 10+ years ago I had to compile it myself. Greg -- Greg Freemyer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 08/12/16 20:27, Greg Freemyer wrote:
On Thu, Dec 8, 2016 at 11:23 AM, Anton Aylward <opensuse@antonaylward.com> wrote:
Yes 'find' + 'cpio' can be very flexible!
Check the man page for 'rmt'. If you're going to/from tape, add buffer or mbuffer to the toolbox. It sits between cpio and the tape drive and helps ensure data is fed to the tape in appropriate size chunks.
Of the 2, I like mbuffer: http://www.maier-komor.de/mbuffer.html
The last source tarball is from just last month, so it is still maintained:
http://www.maier-komor.de/software/mbuffer/mbuffer-20161115.tgz
And it is part of openSUSE for at least the last 7 or 8 years. 10+ years ago I had to compile it myself.
Greg -- Greg Freemyer
Hi All, I appear to have solved it this time! I found a workaround from this page (answer #40) http://stackoverflow.com/questions/9084257/bash-array-with-spaces-in-element... I have my array (my ISOs in this case): files[0]="/srv/media/PS2 Games/" files[1]="/srv/media/PS1 Games/" then I have my loop: file_list="" #use variable $file_list for file in ${files[@]}; do file_list="$file_list \"$file\"" done specifically I replaced: for file in ${files[@]}; do with: for file in "${files[@]}"; do Without the quotes as above the output of the loop ($file_list) was this: (treating the spaces as separate array items): "/srv/media/PS2" "Games/" "/srv/media/PS1" "Games/" Now the quotes are added I get as I expected: "/srv/media/PS2 Games/" "/srv/media/PS1 Games/" It was very strange though because if I put in a file name instead of a directory the problem did not seem to affect the files with spaces?? Maybe a bug or just a quirk... I have just set off a test backup and will see if it makes it to the end OK. P.S. Talking about bash on an off-topic but interesting/worrying note, I just re-installed a mac-book for someone onto Mac OS Sierra (downloaded from the app-store on Wednesday) and the version of bash supplied in the image is still version 4.3 before the "shellshock" bug was patched (over 2 years out of date!!). (Made me laugh/cringe a little :P). Although it did immediately update to the latest Version once I connected to the WiFi. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/09/2016 03:17 PM, Paul Groves wrote:
I appear to have solved it this time! I found a workaround from this page (answer #40) http://stackoverflow.com/questions/9084257/bash-array-with-spaces-in-element...
I have my array (my ISOs in this case): files[0]="/srv/media/PS2 Games/" files[1]="/srv/media/PS1 Games/"
then I have my loop: file_list="" #use variable $file_list for file in ${files[@]}; do file_list="$file_list \"$file\"" done
specifically I replaced: for file in ${files[@]}; do with: for file in "${files[@]}"; do
Without the quotes as above the output of the loop ($file_list) was this: (treating the spaces as separate array items): "/srv/media/PS2" "Games/" "/srv/media/PS1" "Games/"
Now the quotes are added I get as I expected: "/srv/media/PS2 Games/" "/srv/media/PS1 Games/"
With the shell, there are usually many ways to achive a goal. I personally avoid using array shell variables. In this case, I'd probably have used a pipe with xargs. printf '%s\0' \ '/dir/normal/' \ '/dir/with spaces/' \ '/dir/with newline/' \ | xargs -0 tar ... Or I'd have used tar's -files-from= option. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
09.12.2016 18:29, Bernhard Voelker пишет:
On 12/09/2016 03:17 PM, Paul Groves wrote:
I appear to have solved it this time! I found a workaround from this page (answer #40) http://stackoverflow.com/questions/9084257/bash-array-with-spaces-in-element...
I have my array (my ISOs in this case): files[0]="/srv/media/PS2 Games/" files[1]="/srv/media/PS1 Games/"
then I have my loop: file_list="" #use variable $file_list for file in ${files[@]}; do file_list="$file_list \"$file\"" done
specifically I replaced: for file in ${files[@]}; do with: for file in "${files[@]}"; do
Without the quotes as above the output of the loop ($file_list) was this: (treating the spaces as separate array items): "/srv/media/PS2" "Games/" "/srv/media/PS1" "Games/"
Now the quotes are added I get as I expected: "/srv/media/PS2 Games/" "/srv/media/PS1 Games/"
With the shell, there are usually many ways to achive a goal. I personally avoid using array shell variables. In this case, I'd probably have used a pipe with xargs.
printf '%s\0' \ '/dir/normal/' \ '/dir/with spaces/' \ '/dir/with newline/' \ | xargs -0 tar ...
In this example xargs is entirely redundant. And in any more sophisticated case where list is built at run time this does not work without arrays; and using arrays makes xargs redundant again.
Or I'd have used tar's -files-from= option.
Have a nice day, Berny
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Friday 09 December 2016, Paul Groves wrote:
On 08/12/16 20:27, Greg Freemyer wrote:
I have my array (my ISOs in this case): files[0]="/srv/media/PS2 Games/" files[1]="/srv/media/PS1 Games/"
then I have my loop: file_list="" #use variable $file_list for file in ${files[@]}; do file_list="$file_list \"$file\"" done
Your method will not work reliable. Use "${files[@]}" (including the quotes) instead of your file_list as parameter to tar. Have another look at the sample I gave yesterday and see that the "touch" creates the files exactly like they were specified as the individual list elements. You will not get your method working reliable, especially if something like quotes (") are part of the filename, as you throw away the semantics the list gives you (clearly separate members and not something which needs interpretation in one big concatenated string where boundaries again need to be found reliable). BTW: With files[1]="1" files[2]="2a'2b" files[3]=" 3 \" 4 " files[4]="5" files[5]="6 7" files[6]='"8 9"' as files your method produces the files: with 'touch $file_list': """ ""8" "1" "2a'2b" "3" "4" "5" "6" "7" "9"" or with 'touch "$file_list"' the single file "1" "2a'2b" "3" """ "4" "5" "6" "7" ""8" "9"" as output of the touch command (including ALL the quotes). touch simply creates the individual members it received on the command line. Andreas -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hello, On Fri, 09 Dec 2016, Paul Groves wrote:
I have my array (my ISOs in this case): files[0]="/srv/media/PS2 Games/" files[1]="/srv/media/PS1 Games/"
then I have my loop: file_list="" #use variable $file_list for file in ${files[@]}; do file_list="$file_list \"$file\"" done
specifically I replaced: for file in ${files[@]}; do with: for file in "${files[@]}"; do
That should be: file_list=( "${files[@]}" ) or just use "${files[@]}" directly. And to construct the files array, use files=( "foo" "bar baz" "PS1 Games" "PS2 Games" '...' ) I recommend some basic reading, e.g.: http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html http://tldp.org/LDP/Bash-Beginners-Guide/html/index.html http://tldp.org/LDP/abs/html/index.html (see also http://tldp.org/guides.html#abs and http://tldp.org/guides.html#bbg for more formats, e.g. pdf) and specifically on arrays: http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_10_02.html http://tldp.org/LDP/abs/html/arrays.html HTH, -dnh -- printk(KERN_WARNING "Multi-volume CD somehow got mounted.\n"); linux-2.2.16/fs/isofs/inode.c -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (7)
-
Andreas Pfaller -
Andrei Borzenkov -
Anton Aylward -
Bernhard Voelker -
David Haller -
Greg Freemyer -
Paul Groves