Hi Per, On 21/08/14 17:04, Per Jessen wrote:

Hylton Conacher (ZR1HPC) wrote:

...

Hi,

I have a production system that runs OpenSuSe 11.2 that currently acts as a fileserver, mail gateway and workstation. This same production system is on older hardware and thus cannot boot from USB, but can from DVD/CD. The version of dd in Coreutils is 7.1.

My aim is to to upgrade this software to to the latest and greatest the hardware will allow, however before doing a fresh install, I need to preserve the user file and mail server stores i.e. non root users.

The entire system is under 2GB and given I have a a 2TB Drobo storage unit attached via USB that is owned by root, I assume this to be possible.

With the insertion of a Live CD to detect the system and attached USB connected drive array(Drobo), could I use dd to copy the system across and preserve all user permissions, users, groups etc?

Hylton, you will want to use tar or cpio, not dd.

Why do you say to not use dd? I figured that would be the easiest as it seems to be the most simple command. That said however, I do suppose though that using tar or cpio and having one entire file is easier to move around, although without compression, HORRIBLY LARGE. I have just double checked the size of /home directory and at 97GB, that is one MONSTER file so creating an archive is not advisable I would think. So I am not looking at creating and archive file, merely an exact replica to enable bare-metal restore, if necessary, that I can write to a USB attached HDD device of 2TB owned by root. I guess cp would work, but take AGES. My rsync version is 3.0.6 and SSH is not set-up so using rsync is apparently not possible, although given its a copy from a machine to a locally attached hdd, there is no security needed. Given its a local copy why is SSH a factor? So I'm stuck. How to not lose any data or set-up and configuration files but still do a fresh install of 13.1. Those set-up and config files I assume are in /etc and might be needed I'm thinking more and more of rsync but think I would need to possibly change the ownership of the locally attached USB hdd to the same user as whom I want to backup. Help! I'm open to suggestions to accomplish what I seek, if possible. Regards Hylton -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/22/2014 09:18 PM, Per Jessen wrote:

'dd' is the right answer for that, but you need _identical_ hardware to restore to.

Why, would you please elaborate? I mean it's clear that the target device has to be as big or bigger than the source device, but that's it.

I doubt if very many poeple use 'dd' for backups [...]

I can't claim that I'm "many" ... but I did - several times. I'm using it especially when moving/cloning Windows partitions, because it makes a 100% copy of the original. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/23/2014 09:32 AM, Per Jessen wrote:

moving/cloning != backup

right. The use case I meant was more like a poor man's snapshot of a physical machine: copying everything 1:1, trying something completely weird, and then restore the snapshot if necessary. ;-) Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Sat, Aug 23, 2014 at 3:32 AM, Per Jessen <per@computer.org> wrote:

Bernhard Voelker wrote:

...

On 08/22/2014 09:18 PM, Per Jessen wrote:

...

I doubt if very many poeple use 'dd' for backups [...]

I can't claim that I'm "many" ... but I did - several times. I'm using it especially when moving/cloning Windows partitions, because it makes a 100% copy of the original.

Oh, there are a good few purposes for dd, but moving/cloning != backup, IMO.

dd is the core of computer forensic evidence preservation. Used to get full physical images (backups / copies) of drives. It is used in both legal situations where a full copy of the potential evidence must be preserved and in incident response situations where a external hack is suspected. For instance, if you want to do an intense malware investigation, you want to make sure you get every possible sector into your image to make sure you have all the possible remnants at hand at the start of your investigation. The classic 20-year ago approach was: dd if=/dev/sdx of=image_file conv=sync,noerror Today that is not used so much because of the poor bad block handling of dd*. My personal preference is ewfacquire which I've recommended in various dd threads. ewfacquire does allow a "raw" copy to be made which would be the same what dd does. It also allows various forenicic image formats to be used. Those tend to have embedded metadata that speaks to the verification and chain-of-custody. Anyway, I have likely make 100 ewfacquire style disk backups in the last year. It is why I maintain if for the opensuse distro. *The poor bad block handling of dd can be seen by how this command treats bad blocks: dd if=/dev/sda of=image_file bs=1M conv=noerror,sync If on any given 1 megabyte block a media read error is experienced in the middle of the 1 MB block, then dd will fill that sector and the tail of the block after the media error with nulls (or zeros). Thus a media error in the first sector of a 1MB block would cause the failure to preserve the full 1MB of data even though only 512 bytes were failed. Using a smaller blocksize helps, but the reality is that dd as it currently exists (to my knowledge) is not designed to operate optimally in the presence of real world media errors. ewfacquire on the other hand can to be told to read those same 1MB blocks, but if media errors occur to retry the read with ever smaller reads until only the bad sectors actually fail to be copied. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Sat, Aug 23, 2014 at 11:20 AM, Carlos E. R. <robin.listas@gmail.com> wrote:

On 2014-08-23 16:41, Greg Freemyer wrote:

...

Using a smaller blocksize helps, but the reality is that dd as it currently exists (to my knowledge) is not designed to operate optimally in the presence of real world media errors. ewfacquire on the other hand can to be told to read those same 1MB blocks, but if media errors occur to retry the read with ever smaller reads until only the bad sectors actually fail to be copied.

That's what both ddrescue and dd_rhelp (w. dd_rescue) do :-)

Agreed, but for whatever reason I've only used them if my normal tools fail. (ie. ewfacquire and/or dd). I captured an old Red Hat server a couple months ago. I couldn't shut it down and it only had USB-1 ports. I didn't want to install any software so I used "dd if=/dev/sda conv=sync,noerror | netcat" on the server. Then "netcat | ewfacquirestream" on my laptop. You never know what tools you will have to use on a given job. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Bernhard Voelker wrote:

On 08/22/2014 09:18 PM, Per Jessen wrote:

...

'dd' is the right answer for that, but you need _identical_ hardware to restore to.

Why, would you please elaborate? I mean it's clear that the target device has to be as big or bigger than the source device, but that's it.

...

I doubt if very many poeple use 'dd' for backups [...]

I can't claim that I'm "many" ... but I did - several times. I'm using it especially when moving/cloning Windows partitions, because it makes a 100% copy of the original.

Yes, but using dd that's the LEAST DESIRABLE METHOD. We use it with Windows partitions because there is no better choice... FOR WINDOWS partitions. 1: You can't copy a 100GB partition to a 150GB partition without WASTING 50 GB of space 2: because it is a byte-for-byte copy at the block device level, you copy the filesystem with ALL of the fragmentation totally preseerved. Copying to a larger partition is an EXCELLANT time to have the filesystem not only defragged, but also directories reorganized/compacted in the way that the filesystem driver (ext2/3/4, reiser, xfs, etc.) thinks is best. Basically, copying or rsyncing brings the filesystem back into what the designer(s) thought would be the optimal file layout for efficient operations with continued read/write operations. (files being appended, etc.) Using dd offers none of these benefits.

Have a nice day, Berny

-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, Aug 22, 2014 at 3:08 PM, Hylton Conacher (ZR1HPC) <hylton@conacher.co.za> wrote:

Hi Per,

On 21/08/14 17:04, Per Jessen wrote:

...

Hylton Conacher (ZR1HPC) wrote:

...

Hi,

I have a production system that runs OpenSuSe 11.2 that currently acts as a fileserver, mail gateway and workstation. This same production system is on older hardware and thus cannot boot from USB, but can from DVD/CD. The version of dd in Coreutils is 7.1.

My aim is to to upgrade this software to to the latest and greatest the hardware will allow, however before doing a fresh install, I need to preserve the user file and mail server stores i.e. non root users.

The entire system is under 2GB and given I have a a 2TB Drobo storage unit attached via USB that is owned by root, I assume this to be possible.

With the insertion of a Live CD to detect the system and attached USB connected drive array(Drobo), could I use dd to copy the system across and preserve all user permissions, users, groups etc?

Hylton, you will want to use tar or cpio, not dd.

Why do you say to not use dd?

dd is simply not designed to backup directory trees worth of data. If you want to backup a full partition at the raw physical sector level, it is designed to do that.

I figured that would be the easiest as it seems to be the most simple command. That said however, I do suppose though that using tar or cpio and having one entire file is easier to move around, although without compression, HORRIBLY LARGE.

Making a dd backup of the physical drive and/or partition would be even larger!

I have just double checked the size of /home directory and at 97GB, that is one MONSTER file so creating an archive is not advisable I would think.

So I am not looking at creating and archive file, merely an exact replica to enable bare-metal restore, if necessary, that I can write to a USB attached HDD device of 2TB owned by root. I guess cp would work, but take AGES. My rsync version is 3.0.6 and SSH is not set-up so using rsync is apparently not possible, although given its a copy from a machine to a locally attached hdd, there is no security needed. Given its a local copy why is SSH a factor?

For a pure local copy, rsync should work without SSH requirements. To backup /usr to /mnt/usr (on the external drive), I think this will work: sudo rsync --archive /usr /mnt/usr The only tricky part is knowing if you need a trailing /. ie. the command might be: sudo rsync --archive /usr/ /mnt/usr I always have to experiment with the trailing / everytime I use rsync.

So I'm stuck. How to not lose any data or set-up and configuration files but still do a fresh install of 13.1. Those set-up and config files I assume are in /etc and might be needed

I'm thinking more and more of rsync but think I would need to possibly change the ownership of the locally attached USB hdd to the same user as whom I want to backup.

I'm pretty sure that if you run rsync as root with the --archive flag it will retain the old ownership on the backup. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/22/2014 09:21 PM, Greg Freemyer wrote:

To backup /usr to /mnt/usr (on the external drive), I think this will work:

sudo rsync --archive /usr /mnt/usr

The only tricky part is knowing if you need a trailing /. ie. the command might be:

sudo rsync --archive /usr/ /mnt/usr

I always have to experiment with the trailing / everytime I use rsync.

Same here - almost, because I got used to use the trick to append "/." on both the source and the target side ;-) sudo rsync -Haix /usr/. /mnt/usr/. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/23/2014 02:05 AM, Philip Amadeo Saeli wrote:

3. Including a trailing slash on the source means that only the contents of the source dir, and not the dir itself, will appear in the destination dir. Note that all files, even files beginning with a dot, will be included in this case (which beats using a "*").

Thanks for the explanation, but ... I feel uncomfortable with the latter as adding a trailing slash on a symlink (whether to a file or a directory) usually causes the symlink to be de-referenced. Now what happens then in rsync? Will the slash implicitly be added to the de-referenced symlink target again, or not? Thus, I'll stick with "path/to/dir/." which works whether "dir" is a symlink to a directory or not. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/22/2014 03:08 PM, Hylton Conacher (ZR1HPC) wrote:

Why do you say to not use dd? I figured that would be the easiest as it seems to be the most simple command.

cat is even simpler! That's not the point. The point is using a tool that is suitable for the task.

That said however, I do suppose though that using tar or cpio and having one entire file is easier to move around, although without compression, HORRIBLY LARGE.

*IF* you could dd all of /home that would be large as well.

I have just double checked the size of /home directory and at 97GB, that is one MONSTER file so creating an archive is not advisable I would think.

That is a complete non-sequitor. The size of the archive is the same size as what you are dealing with. Distributions on LiveCD are often a single, compressed file that is mounted using FUSE. That's the whole OS, all binaries and script -- everyt9ing. As a single file. There would be nothing magic about having all of /home as single tar or cpio file.

So I am not looking at creating and archive file, merely an exact replica to enable bare-metal restore, if necessary, that I can write to a USB attached HDD device of 2TB owned by root.

"bare metal"? That would be a dd of the raw disk dd if=/dev/sda of=/dev/useba And I would not advise that either!

I guess cp would work, but take AGES.

Look at the code: while ( readchar() ) writechar() But it *IS* a tree-walker, which dd is not.

My rsync version is 3.0.6 and SSH is not set-up so using rsync is apparently not possible, although given its a copy from a machine to a locally attached hdd, there is no security needed. Given its a local copy why is SSH a factor?

I've been using rsync for eyars for disk-to-disk copy WITHOUT ssh SSH is *NOT* repeat *NOT* required for disk-to-disk copy on the same machine. Its very clearly stated in the man page. Lots of example of that use rsync -av /src/foo /dest rsync -av /src/foo/ /dest/foo

So I'm stuck. How to not lose any data or set-up and configuration files but still do a fresh install of 13.1. Those set-up and config files I assume are in /etc and might be needed

Most of us do this be using rsync to save /home and whatever parts of /var might be relevant such as mail, dns, dhcp, crom ... A lot of what appears to be in /etc isn't, its actually symlinks and you need to save the target. Much of what _should_ be in the new /etc is generated by the install. Think hard about what parts of /etc you really want and what needs to be regenerated for the new system. Merely saving and restring /etc wholesale will break things. I can't tell you what, I can only tell you that you need to inspect and think about it.

I'm thinking more and more of rsync but think I would need to possibly change the ownership of the locally attached USB hdd to the same user as whom I want to backup.

No. If you do this as root at use the rsync flags to preserve that, or use the tar or cpio to preserve that, its a non-issue. All this is old hat, We've been doing it for more than a quarter of a century. We may not - most of us - be using tape any more, but the TAR format is still in use since 'tar' is a treewalker and preserves all the ownership & permission information. Most of your fears/objections are groundless and a RTFM should make that clear.

Help! I'm open to suggestions to accomplish what I seek, if possible.

${DEITY}-${EXPLETIVE} man, just get on with using rsync. This isn't rocket science; its not even computing science, it just a backup/restore. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/22/2014 10:52 PM, Dirk Gently wrote:

It is virtually unchanged since the 1960's [...]

According to the version control system [1], GNU coreutils' version of dd(1) is from Oct 1992 and had 287 changes since then. $ git log --oneline src/dd.c | wc -l 287 [1] http://git.savannah.gnu.org/cgit/coreutils.git/log/src/dd.c

[...] -- NO CODE has ever been added throughout the last 45 years for dd to understand filesystems, or directory structures, or to give it the ability to walk through a directory structure.

Of course, because file system structures are completely beyond dd's domain; it's like asking "why can't dd(1) do 1+1?". It's not a calculator. The discussion is therefore quite "fruitful": apples and oranges. ;-) Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/25/2014 03:24 PM, Dirk Gently wrote:

I'm talking about adding significant algorithms, not minor tweaks to system calls to take advantage of new services and efficiencies added to the kernel.

And your cite... 1992.. the year is now 2014... that's 22 YEARS without even a tweak.

For a tool whose task [1] is to "Convert and copy a file" (among all implementations), what would you suggest it to do more than today (without bloating the code with functionality already possible with or in combination with other tools)? [1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html Regarding the strange synopsis: as you said dd(1) is very old, and all implementations follow the given rule. Therefore, it's simply too late to change how options and operands are specified. It's hard enough to kill obsolete uses like e.g. "tail -5" (correctly nowadays: "tail -n 5") which took almost a decade, so changing all of the options in dd.c is impossible. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/25/2014 09:18 PM, Dirk Gently wrote:

Bernhard Voelker wrote:

...

It's hard enough to kill obsolete uses like e.g. "tail -5" (correctly nowadays: "tail -n 5") which took almost a decade, so changing all

The syntax tail -5 was never removed.

Agreed, not removed, but it is a deprecated, obsolete syntax. It is highly recommended to change it to the newer form. It still was part of the specification back in 1997: Single UNIX Specification, Version 2, 1997 http://pubs.opengroup.org/onlinepubs/7908799/xcu/tail.html which has been removed in Issue 6 http://pubs.opengroup.org/onlinepubs/009695399/utilities/tail.html and "Issue 7" of the "IEEE Std 1003.1", 2013 Edition, now contains a word about that it's non-standard: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/tail.html Earlier versions of this standard allowed the following forms in the SYNOPSIS: tail -[number][b|c|l][f] [file]tail +[number][b|c|l][f] [file] These forms are no longer specified by POSIX.1-2008, but may be present in some implementations. ... and the state is also documented in the texinfo manual: http://www.gnu.org/software/coreutils/manual/html_node/tail-invocation.html For compatibility tail also supports an obsolete usage ‘tail -[count][bcl][f] [file]’, which is recognized only if it does not conflict with the usage described above. This obsolete form uses exactly one option and at most one file. However, the syntax "tail +5 file" has already gone http://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Old-tail-plus-N... compare $ env _POSIX2_VERSION=199201 /usr/bin/tail +1 /etc/passwd vs. $ /usr/bin/tail +1 /etc/passwd

Read the man page for dd for Version 7 Unix (circa 1977). dd is not substantially changed.

The ENTIRE Version 7 manual (both Volume 1 and Volume 2) is available here.

The "man' command looks up things in volume 1, and you will find the dd command in this document http://cm.bell-labs.com/7thEdMan/v7vol1.pdf

Thanks for the URLs. Compared to that old document, dd(1) has got the iflag=/oflag= and the status arguments, various conv= modes like block, unblock, sparse, excl, nocreat, notrunc, noerror, fdatasync and fsync, and of course support for specifying the numbers with suffixes K, M, G, T, P, E, Z, Y. Not that bad. ;-)

DD comes from IBM mainframes.

Nobody disputes this except for you.

I never did. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Hi Anton, On 21/08/14 19:01, Anton Aylward wrote:

On 08/21/2014 12:26 PM, Greg Freemyer wrote:

...

Good Luck and if you find a tutorial that addresses a use case of using dd to copy directories, please post a link here for all of our education.

Indeed. dd is not a file tree walker like rsync, tar find|cpio

Ahhh, thank you for sharing 'your' scars. I did not know this.

Hylton, I use rsync to make file system images/backup such as you describe.

Pray tell a cli command writing / to a root owned device /drobo but preserving all source perms and file tree structure, and of course excluding the destination as a source as it is on the root filesystem? Regards Hylton -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/22/2014 03:16 PM, Hylton Conacher (ZR1HPC) wrote:

Hi Anton,

On 21/08/14 19:01, Anton Aylward wrote:

...

On 08/21/2014 12:26 PM, Greg Freemyer wrote:

...

Good Luck and if you find a tutorial that addresses a use case of using dd to copy directories, please post a link here for all of our education.

Indeed. dd is not a file tree walker like rsync, tar find|cpio

Ahhh, thank you for sharing 'your' scars. I did not know this.

...

Hylton, I use rsync to make file system images/backup such as you describe.

Pray tell a cli command writing / to a root owned device /drobo but preserving all source perms and file tree structure, and of course excluding the destination as a source as it is on the root filesystem?

You don't want to do that in a lot of cases. That will give you a full backup. If you are backing up your drive, throwing it away and getting a new drive, then installing a new OS then yes, you will want to back up a lot, but not like that. You new install of 13.1 will have new /dev, /sys/ as well as libraries and libraries. Most of us upgrading in place have /home (as well as other thing we want to survive an upgrade) in separate partitions. In the upgrade we simply tell the installer not to reformat those partitions. Hence /home, mail, web site and more make it though. But if you are installing to a virgin disk, think about what that partitioning and set it up like that so that things are simpler & easier in the future. When you decide what specifics are to survive from your soon-to-be-trashed drive then back them up. I use LVM and have things arranged in 5G partitions since they back up easily ont a DVD. Sadly rsync won't write to a DVD but k3b does that just fine. I have partitions under my /home/anton, so I never get your huge file. I have ~/Document, and that's divided down as well; ~/Photographs by year and month and special event. That's a lot easier as much never changes. Once I have a baseline backup to a collection of DVDs I can run incremental; using 'find' to tell me what's changed since the last backup. None of this technique is new. I was doing with with PDP-11s back in the late 1970s on V6 and V7 UNIX to 1600bpi tapes. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/21/2014 03:55 PM, Greg Freemyer wrote:

I opened a bugzilla

https://bugzilla.novell.com/show_bug.cgi?id=893036

Basically it says either the man page is wrong or examples of how to use these args need to be added.

The best I could get to was:

dd if=tmp/ iflag=directory,noatime,nofollow of=tmp1/ oflag=directory

dd: failed to open tmp1: Is a directory.

At least that didn't give me a syntax error. What it should do I don't know.

I think this is wrong-headed, misguided. The man page says DD is to copy a file, not walk a directory tree. If you want to treat a directory as file, OK, but its an array of byes not a container. At best, I think that your /tmp -> tmp1 will give you another directory with what amounts to the inodes&names of the first. Not hard links, and you'll have screwed up one, possibly two file systems, Run FSCK ASAP. DD is most certainly NOT a tree walker. There are other file operations you can perform that give the same info as dd. Try cat /tmp tr /tmp od -cb /tmp Like dd these are all FILE operations. All give the same error: "Is a directory" That's what you get when you try running a file tool on a directory. One might think that this is meaningful dd if=/tmp iflag=directory,noatime,nofollow | od -cb Try it. I think that "iflag=directory" doesn't mean what you think it means! -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Anton Aylward wrote:

DD is most certainly NOT a tree walker.

Yup. From the top of the man page: "dd - convert and copy a file". -- Per Jessen, Zürich (13.6°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, Aug 22, 2014 at 6:22 AM, Bernhard Voelker <mail@bernhard-voelker.de> wrote:

On 08/21/2014 10:38 PM, Anton Aylward wrote:

...

Try

cat /tmp tr /tmp od -cb /tmp

Like dd these are all FILE operations. All give the same error: "Is a directory" That's what you get when you try running a file tool on a directory.

And that's the point: it's not a limitation in the tool - whatever the tool would do with such data - but rather a limit implied by the kernel. And the info page states this clearly:

`directory' Fail unless the file is a directory. Most operating systems do not allow I/O to a directory, so this flag has limited utility.

You can observe it with 'strace -v dd if=/tmp iflag=directory':

... open("/tmp", O_RDONLY|O_DIRECTORY) = 3 dup2(3, 0) = 0 close(3) = 0 lseek(0, 0, SEEK_CUR) = 0 ... read(0, 0x8e5000, 512) = -1 EISDIR (Is a directory)

If I remember correctly from +10 years ago, 'cat /some/dir' works on e.g. HP-UX, although the use is quite limited.

Have a nice day, Berny

Berny, Why does dd have a flag option of nofollow? Why noatime? I've never noticed either of them before this discussion, but they seem foreign to a tool like dd. nofollow in particular seems crazy. If I don't want dd to work if "if" points at a symlink, then I shouldn't be calling dd on it in the first time. As noatime, that's a great option for a tree walking backup tool, but why in world does it make sense for dd. The above plus the "directory" flag make me think that ther is a use case for dd that I've never considered let alone used. Something like a shell script that can walk a directory tree and call dd on each of the files in order to create a backup copy. I can conceive of that but the problems of metadata data handling and non-files (eg. devices, sockets, symlinks, etc) seem to overwhelm the use case. My argument is that without an legitimate elucidation of the use case for "directory", "nofollow", and "noatime" they should be removed from dd and from the documentation for dd. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, Aug 22, 2014 at 11:18 AM, Per Jessen <per@computer.org> wrote:

Greg Freemyer wrote:

...

On Fri, Aug 22, 2014 at 6:22 AM, Bernhard Voelker <mail@bernhard-voelker.de> wrote:

...

On 08/21/2014 10:38 PM, Anton Aylward wrote:

...

Try

cat /tmp tr /tmp od -cb /tmp

Like dd these are all FILE operations. All give the same error: "Is a directory" That's what you get when you try running a file tool on a directory.

And that's the point: it's not a limitation in the tool - whatever the tool would do with such data - but rather a limit implied by the kernel. And the info page states this clearly:

`directory' Fail unless the file is a directory. Most operating systems do not allow I/O to a directory, so this flag has limited utility.

You can observe it with 'strace -v dd if=/tmp iflag=directory':

... open("/tmp", O_RDONLY|O_DIRECTORY) = 3 dup2(3, 0) = 0 close(3) = 0 lseek(0, 0, SEEK_CUR) = 0 ... read(0, 0x8e5000, 512) = -1 EISDIR (Is a directory)

If I remember correctly from +10 years ago, 'cat /some/dir' works on e.g. HP-UX, although the use is quite limited.

Have a nice day, Berny

Berny,

Why does dd have a flag option of nofollow? Why noatime? nofollow in particular seems crazy. If I don't want dd to work if "if" points at a symlink, then I shouldn't be calling dd on it in the first time.

Maybe if you're using dd in a script or loop, maybe in connection with find or some such.

...

My argument is that without an legitimate elucidation of the use case for "directory", "nofollow", and "noatime" they should be removed from dd and from the documentation for dd.

One could argue that their presence hasn't caused many problems sofar :-)

Well, they caused this entire thread because someone came to the conclusion that dd was a tree walker since it had arguments that seemingly only make sense in a tree walker. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, Aug 22, 2014 at 1:13 PM, Per Jessen <per@computer.org> wrote:

Greg Freemyer wrote:

...

On Fri, Aug 22, 2014 at 11:18 AM, Per Jessen <per@computer.org> wrote:

...

Greg Freemyer wrote:

...

On Fri, Aug 22, 2014 at 6:22 AM, Bernhard Voelker <mail@bernhard-voelker.de> wrote:

...

On 08/21/2014 10:38 PM, Anton Aylward wrote:

...

Try

cat /tmp tr /tmp od -cb /tmp

Like dd these are all FILE operations. All give the same error: "Is a directory" That's what you get when you try running a file tool on a directory.

And that's the point: it's not a limitation in the tool - whatever the tool would do with such data - but rather a limit implied by the kernel. And the info page states this clearly:

`directory' Fail unless the file is a directory. Most operating systems do not allow I/O to a directory, so this flag has limited utility.

You can observe it with 'strace -v dd if=/tmp iflag=directory':

... open("/tmp", O_RDONLY|O_DIRECTORY) = 3 dup2(3, 0) = 0 close(3) = 0 lseek(0, 0, SEEK_CUR) = 0 ... read(0, 0x8e5000, 512) = -1 EISDIR (Is a directory)

If I remember correctly from +10 years ago, 'cat /some/dir' works on e.g. HP-UX, although the use is quite limited.

Have a nice day, Berny

Berny,

Why does dd have a flag option of nofollow? Why noatime? nofollow in particular seems crazy. If I don't want dd to work if "if" points at a symlink, then I shouldn't be calling dd on it in the first time.

Maybe if you're using dd in a script or loop, maybe in connection with find or some such.

...

My argument is that without an legitimate elucidation of the use case for "directory", "nofollow", and "noatime" they should be removed from dd and from the documentation for dd.

One could argue that their presence hasn't caused many problems sofar :-)

Well, they caused this entire thread because someone came to the conclusion that dd was a tree walker since it had arguments that seemingly only make sense in a tree walker.

Right, but if that's perhaps the only "problem" caused in all the many years of dd supporting those flags, I would personally not call it worth fixing. In fact, because 'dd' has had those flags for years, "fixing" it sounds like a decidedly bad idea. We may not know what the use cases are, but if we were to fix the flags, I suspect we would find out in about a nanosecond :-)

Agreed that at least in linux, the commands have been around at least since 2010. Here's one example: http://linux.die.net/man/1/dd But counter examples where they don't exist are easy to find. Here's a few: This 10 year old man page doesn't have any of the iflag/oflag capability: http://www.linuxmanpages.com/man1/dd.1.php Neither does the 2013 dd man page from freebsd http://www.freebsd.org/cgi/man.cgi?dd(1) How about UNIX. No, as of 2012 this man page doesn't have any iflag/oflag option: http://unixhelp.ed.ac.uk/CGI/man-cgi?dd So the only place I have yet found that even allows the "directory" flag is linux, but the linux kernel blocks it from working because it doesn't provide read or write access to raw directories. If I ever saw a feature that was broken by design this seems to be it. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/22/2014 01:28 PM, Greg Freemyer wrote:

So the only place I have yet found that even allows the "directory" flag is linux, but the linux kernel blocks it from working because it doesn't provide read or write access to raw directories. If I ever saw a feature that was broken by design this seems to be it.

Indeed. The argument processing for this http://www.opensource.apple.com/source/file_cmds/file_cmds-220.7/dd/args.c apples out of Berkeley, doesn't seem to mention 'directory" -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

* Greg Freemyer <greg.freemyer@gmail.com> [2014-08-22 15:32]:

On Fri, Aug 22, 2014 at 3:59 PM, Anton Aylward <opensuse@antonaylward.com> wrote:

...

On 08/22/2014 01:28 PM, Greg Freemyer wrote:

...

So the only place I have yet found that even allows the "directory" flag is linux, but the linux kernel blocks it from working because it doesn't provide read or write access to raw directories. If I ever saw a feature that was broken by design this seems to be it.

Indeed. The argument processing for this http://www.opensource.apple.com/source/file_cmds/file_cmds-220.7/dd/args.c apples out of Berkeley, doesn't seem to mention 'directory"

That looks to be a totally different source tree than openSUSE is using.

[snip]

It looks like the only use of "flag=directory,no_atime,no_follow" is to set the 3 flags:

O_DIRECTORY O_NOATIME O_NOFOLLOW

which are in turn passed to fd_reopen(..)

From open(2):

O_DIRECTORY If pathname is not a directory, cause the open to fail. This flag is Linux-specific, and was added in kernel version 2.1.126, to avoid denial-of-service problems if opendir(3) is called on a FIFO or tape device. --Phil -- Philip Amadeo Saeli openSUSE, CentOS, RHEL psaeli@zorodyne.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, Aug 22, 2014 at 6:15 PM, Bernhard Voelker <mail@bernhard-voelker.de> wrote:

On 08/22/2014 07:28 PM, Greg Freemyer wrote:

...

So the only place I have yet found that even allows the "directory" flag is linux, but the linux kernel blocks it from working because it doesn't provide read or write access to raw directories.

Well, the Linux kernel is not the only OS for GNU coreutils.

Why do I suspect O_DIRECTORY is a linux only flag for open. In the man page for open it says: -- O_DIRECTORY - If pathname is not a directory, cause the open to fail. This flag is Linux-specific, and was added in kernel version 2.1.126, to avoid denial-of-service problems if opendir(3)is called on a FIFO or tape device, but should not be used outside of the implementation of opendir(3). -- I'm going to drop out of this thread since it's not going anywhere, but I really suspect the "directory" flag was added to dd by some over-zealous developer. It clearly doesn't have a use-case because it is not meant for applications to ever open directories other than via opendir(3). Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, Aug 22, 2014 at 7:50 PM, Bernhard Voelker <mail@bernhard-voelker.de> wrote:

On 08/23/2014 01:39 AM, Greg Freemyer wrote:

...

O_DIRECTORY - If pathname is not a directory, cause the open to fail. This flag is Linux-specific, and was added in kernel version 2.1.126, to avoid denial-of-service problems if opendir(3)is called on a FIFO or tape device, but should not be used outside of the implementation of opendir(3).

Interesting, "man 2 open" lacks the latter ", but should not ..." sentence my 13.1 system.

I was on a 13.1 server with no man pages installed, so I quoted from: http://linux.die.net/man/2/open -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/22/2014 11:13 AM, Greg Freemyer wrote:

My argument is that without an legitimate elucidation of the use case for "directory", "nofollow", and "noatime" they should be removed from dd and from the documentation for dd.

Greg

What does the code say? What does the code do with these flags? "Use the <strike>F</strike>Source, <strike>Luke</strike> Greg". -- /"\ \ / ASCII Ribbon Campaign X Against HTML Mail / \ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Greg Freemyer wrote:

On Fri, Aug 22, 2014 at 6:22 AM, Bernhard Voelker <mail@bernhard-voelker.de> wrote:

...

On 08/21/2014 10:38 PM, Anton Aylward wrote:

...

Try

cat /tmp tr /tmp od -cb /tmp

Like dd these are all FILE operations. All give the same error: "Is a directory" That's what you get when you try running a file tool on a directory.

And that's the point: it's not a limitation in the tool - whatever the tool would do with such data - but rather a limit implied by the kernel. And the info page states this clearly:

`directory' Fail unless the file is a directory. Most operating systems do not allow I/O to a directory, so this flag has limited utility.

You can observe it with 'strace -v dd if=/tmp iflag=directory':

... open("/tmp", O_RDONLY|O_DIRECTORY) = 3 dup2(3, 0) = 0 close(3) = 0 lseek(0, 0, SEEK_CUR) = 0 ... read(0, 0x8e5000, 512) = -1 EISDIR (Is a directory)

If I remember correctly from +10 years ago, 'cat /some/dir' works on e.g. HP-UX, although the use is quite limited.

Have a nice day, Berny

Berny,

Why does dd have a flag option of nofollow? Why noatime?

I've never noticed either of them before this discussion, but they seem foreign to a tool like dd.

nofollow in particular seems crazy. If I don't want dd to work if "if" points at a symlink, then I shouldn't be calling dd on it in the first time.

As noatime, that's a great option for a tree walking backup tool, but why in world does it make sense for dd.

The above plus the "directory" flag make me think that ther is a use case for dd that I've never considered let alone used.

Something like a shell script that can walk a directory tree and call dd on each of the files in order to create a backup copy. I can conceive of that but the problems of metadata data handling and non-files (eg. devices, sockets, symlinks, etc) seem to overwhelm the use case.

My argument is that without an legitimate elucidation of the use case for "directory", "nofollow", and "noatime" they should be removed from dd and from the documentation for dd.

They make it easier to write shell scripts that call dd within some sort of tree-walker such as find(1)

Greg

-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/22/2014 05:13 PM, Greg Freemyer wrote:

Why does dd have a flag option of nofollow? Why noatime?

I've never noticed either of them before this discussion, but they seem foreign to a tool like dd.

nofollow in particular seems crazy. If I don't want dd to work if "if" points at a symlink, then I shouldn't be calling dd on it in the first time.

As noatime, that's a great option for a tree walking backup tool, but why in world does it make sense for dd.

Well, the difference is that it opens the input file differently: open("x", O_RDONLY|O_NOATIME) = 3 vs. open("x", O_RDONLY) = 3 POSIX [1] doesn't specify 'iflag' at all, so this is a GNU extension. [1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html 'noatime' and 'nofollow' support has been added in 2005/2006 in version v6.0 without long commit message. But as you know from other flags, dd(1) is quite a low-level-I/O user- land tool, so it has many fine-grained options which are directly passed to low-level glibc functions like read(2), open(2), write(2), etc.

My argument is that without an legitimate elucidation of the use case for "directory", "nofollow", and "noatime" they should be removed from dd and from the documentation for dd.

Well, 'directory' may be helpful on systems that support it. 'nofollow' is a nice trick to prevent reading thru a symlink, and 'noatime' is just another possible flag used by open(2). dd(1) is a sophisticated Swiss-army knife with many low-level knobs. Admittedly, the texinfo manual could have more examples, yet the sheer number of combinations would make it impossible to add an example for every use case. Of course, everyone is invited to make proposals to enhance the docs, and I will try to carry them to upstream if possible. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/22/2014 11:05 PM, Dirk Gently wrote:

on all the old unix systems, you could not only cat a directory, but even open them up in vi (not recommended!)

true, and nowadays, VIM automatically shows a directory listing if you do "vi /". Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Greg Freemyer wrote:

On Thu, Aug 21, 2014 at 12:26 PM, Greg Freemyer <greg.freemyer@gmail.com> wrote:

...

On Thu, Aug 21, 2014 at 10:16 AM, Hylton Conacher (ZR1HPC) <hylton@conacher.co.za> wrote:

...

Hi,

I have a production system that runs OpenSuSe 11.2 that currently acts as a fileserver, mail gateway and workstation. This same production system is on older hardware and thus cannot boot from USB, but can from DVD/CD. The version of dd in Coreutils is 7.1.

My aim is to to upgrade this software to to the latest and greatest the hardware will allow, however before doing a fresh install, I need to preserve the user file and mail server stores i.e. non root users.

The entire system is under 2GB and given I have a a 2TB Drobo storage unit attached via USB that is owned by root, I assume this to be possible.

With the insertion of a Live CD to detect the system and attached USB connected drive array(Drobo), could I use dd to copy the system across and preserve all user permissions, users, groups etc?

A command of: $dd if=/ of=/drobo/ fsync noatime nofollow

That would enable me to then upgrade the system, restore the /home and /etc folders without overwriting any new files and then either learn to use kbackup or SSH/rsync etc to implement a proper backup to Drobo.

Possible?, snafu's?, other thoughts?

Regards Hylton

Hylton,

I like to think I'm a dd expert, but I'm not familiar with this type of dd usage at all.

I have never pointed if or of at directories.

I see supported arguments in the man page for directory, noatime and nofollow which just confuses me even more. (With openSUSE 13.1)

I can't figure out a legal syntax to use any of "directory", "noatime", or "nofollow".

I opened a bugzilla

https://bugzilla.novell.com/show_bug.cgi?id=893036

Basically it says either the man page is wrong or examples of how to use these args need to be added.

The best I could get to was:

dd if=tmp/ iflag=directory,noatime,nofollow of=tmp1/ oflag=directory

dd: failed to open tmp1: Is a directory.

At least that didn't give me a syntax error. What it should do I don't know.

Greg

Greg, yOu're doing it wrong. dd is used to copy binary files, or binary objects... you can use it to copy a COMPLETE filesystem (i.e. partition) but not a directory tree within that partition. I have far too much experience with dd. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/22/2014 09:20 PM, Hylton Conacher (ZR1HPC) wrote:

[...] many man cmds need greater expansion [...]

For GNU projects like 'coreutils' - which dd(1) is part of - the way is to keep the man pages terse and to put documentation effort in the Texinfo manual, as suggested at the end of the man page: $ info coreutils 'dd invocation' or online: http://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html Hylton, for your case, dd(1) might not be the right tool, though. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 08/22/2014 03:23 AM, Dirk Gently wrote:

...

If you're going to dump it to an empty disk, use rsync.

+1. No, wait +10^23

This will do several things -- if there are logical block errors on the target disk, the rsync will be more robust. 2: the rsync copy will be fully defragged and spatially compacted (files in the same directory will be together).

All very true But the real point is that rsync is a tree walker whereas dd is there to copy on a single file.

dd will be faster, but the copied image will be of lower quality.

Implied in that is that the image being copied is of the raw disk, a binary blob, and not a sequence of files. It is copied byte for byte including all the sector by sector layout. Three is no compacting, handling of errors is primitive and on the while they will simply be replicated and may even cause dd to abort. Directories will not be compacted. Same with copying back.

dd will take about 30% to 50% as much time as the rsync solution, but when your restore your files to the new installation, the rsynced filesystem can be rsynced back to your new system much faster than rsyncing a dd'ed copy.

And copying back may not work. a dd of a raw disk assumes that the target has similar characteristics to the source. It may work better with a primitive file system than more modern ones, but I'm not sure of even that. IF, perhaps, you were dd'ing a raw disk off to store and then replacing the original source with an identical disk, then I'd have a degree of confidence. I said that dd is a wrong headed solution here. I stand by that. Whatever Dirk has to say about speed/efficiency, the original idea of backup-restore while replacing a disk is better carried out with something like rsync. All in all I'd recommend rsync over tar or find/cpio since it is more modern and more flexible. I have myself done things similar to what the OP was proposing, so can heartily recommend rsync. Dd is a wonderful tool but its not the tool that should be used here. -- Tortoise: 'How many talking tortoises have you met?' Brutha: 'I don't know.' Tortoise: 'What d'you mean, you don't know?' Brutha: 'Well, they might all talk. They just might not say anything when I'm there.' -- "Small Gods", Terry Pratchett -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org