I used "dd if=/dev/hdb2 of=/dev/another_drive_on_same_machine/image_file" (where hdb2 contained the SuSE 9.1 installation with top directory root or "/" ) and from where I was executing the"dd" command (as root) in order to initially create a complete backup of the disk in the form of an image file located on "another_drive_on_same_machine" so that I could move it to another LAN computer capable of blasting it onto a DVD. Now the "another_drive_on_same_machine" happens to be mounted as /HDD2 in the installation on /dev/hdb2. HDD2 is my drive for backups. It has to be mounted and therefore is also found in the directory path under "/". This of course leads to the problem where "dd" (essentially a file copy utility) includes the content of /HDD2 and there is no "exclude" provision in the "dd" command. I could move the contents out of HDD2 (80 Gig drive) but I do not have an unused drive on the same machine and the drives on the other LAN machines are not able to take this additional load to use as a temporary storage. Any ideas? BTW: I tried DAR but it has been an entire week creating a compressed backup of "/" on /dev/hdb2. The DAR package has some nice features but speed is not one of them nor is an image file. In one week DAR has only created a compressed backup file less then 300Megs. The installation it is working on is approx. 7 Gig uncompressed. By the end of January some time DAR may have completed it's task it it doesn't error out. I don't have a tape drive that works. Does anyone have any ideas how I might create a DVD image file that gets around the problem I am having with "dd" (that is I cannot give it a directory exclusion as "/HDD2"). Thanks in advance for any useful suggestions. TED
On Wed, 29 Dec 2004, Ted Hilts wrote:
I used "dd if=/dev/hdb2 of=/dev/another_drive_on_same_machine/image_file" (where hdb2 contained the SuSE 9.1 installation with top directory root or "/" ) and from where I was executing the"dd" command (as root) in order to initially create a complete backup of the disk in the form of an image file located on "another_drive_on_same_machine" so that I could move it to another LAN computer capable of blasting it onto a DVD.
Now the "another_drive_on_same_machine" happens to be mounted as /HDD2 in the installation on /dev/hdb2. HDD2 is my drive for backups. It has to be mounted and therefore is also found in the directory path under "/".
This of course leads to the problem where "dd" (essentially a file copy utility) includes the content of /HDD2 and there is no "exclude" provision in the "dd" command. I could move the contents out of HDD2 (80 Gig drive) but I do not have an unused drive on the same machine and the drives on the other LAN machines are not able to take this additional load to use as a temporary storage.
1. dd is NOT a file copy utility. In the above scenario, it is being used to copy disk blocks. 2. use rsync. The most common and basic invocation is rsync -av source_dir dest_dir read up on how rsync works if you have trouble rsync can do exclusions. dd is so completely the wrong tool for that job.
Does anyone have any ideas how I might create a DVD image file that gets around the problem I am having with "dd" (that is I cannot give it a directory exclusion as "/HDD2").
Yes. Use tar.
tar cvf name_of_tarball stuff_to_backup --exclude="/excludeme"
Unless you need to don't bother with compression it just slows the
process down.
--
Carpe diem - Seize the day.
Carp in denim - There's a fish in my pants!
Jon Nelson
On Wednesday 29 December 2004 20:05, Jon Nelson wrote:
Ted Hilts wrote:
I used "dd if=/dev/hdb2 of=/dev/another_drive
if you mount another partition at , say, /mnt then, TAR will take about 10 minutes to back up about 5 gig system :- the tar command works fine : .............................. tar clf - . | ( umask 0; cd /mnt; tar xvf - ) ............................................................ more generally, RSYNC is an excellent backup program best rgds ___________
Riccardo Thanks for your input. TED riccardo wrote:
On Wednesday 29 December 2004 20:05, Jon Nelson wrote:
Ted Hilts wrote:
I used "dd if=/dev/hdb2 of=/dev/another_drive
if you mount another partition at , say, /mnt
then, TAR will take about 10 minutes to back up about 5 gig system :-
the tar command works fine : ..............................
tar clf - . | ( umask 0; cd /mnt; tar xvf - )
............................................................
more generally, RSYNC is an excellent backup program
best rgds ___________
Jon Nelson Thanks for you input. TED Jon Nelson wrote:
On Wed, 29 Dec 2004, Ted Hilts wrote:
I used "dd if=/dev/hdb2 of=/dev/another_drive_on_same_machine/image_file" (where hdb2 contained the SuSE 9.1 installation with top directory root or "/" ) and from where I was executing the"dd" command (as root) in order to initially create a complete backup of the disk in the form of an image file located on "another_drive_on_same_machine" so that I could move it to another LAN computer capable of blasting it onto a DVD.
Now the "another_drive_on_same_machine" happens to be mounted as /HDD2 in the installation on /dev/hdb2. HDD2 is my drive for backups. It has to be mounted and therefore is also found in the directory path under "/".
This of course leads to the problem where "dd" (essentially a file copy utility) includes the content of /HDD2 and there is no "exclude" provision in the "dd" command. I could move the contents out of HDD2 (80 Gig drive) but I do not have an unused drive on the same machine and the drives on the other LAN machines are not able to take this additional load to use as a temporary storage.
1. dd is NOT a file copy utility. In the above scenario, it is being used to copy disk blocks.
2. use rsync. The most common and basic invocation is rsync -av source_dir dest_dir
read up on how rsync works if you have trouble rsync can do exclusions.
dd is so completely the wrong tool for that job.
Does anyone have any ideas how I might create a DVD image file that gets around the problem I am having with "dd" (that is I cannot give it a directory exclusion as "/HDD2").
Yes. Use tar.
tar cvf name_of_tarball stuff_to_backup --exclude="/excludeme"
Unless you need to don't bother with compression it just slows the process down.
-- Carpe diem - Seize the day. Carp in denim - There's a fish in my pants!
Jon Nelson
On Wed, 29 Dec 2004 12:57:59 -0700, Ted Hilts
I used "dd if=/dev/hdb2 of=/dev/another_drive_on_same_machine/image_file" (where hdb2 contained the SuSE 9.1 installation with top directory root or "/" ) and from where I was executing the"dd" command (as root) in order to initially create a complete backup of the disk in the form of an image file located on "another_drive_on_same_machine" so that I could move it to another LAN computer capable of blasting it onto a DVD.
Now the "another_drive_on_same_machine" happens to be mounted as /HDD2 in the installation on /dev/hdb2. HDD2 is my drive for backups. It has to be mounted and therefore is also found in the directory path under "/".
This of course leads to the problem where "dd" (essentially a file copy utility) includes the content of /HDD2 and there is no "exclude" provision in the "dd" command. I could move the contents out of HDD2 (80 Gig drive) but I do not have an unused drive on the same machine and the drives on the other LAN machines are not able to take this additional load to use as a temporary storage.
Any ideas?
As others have said, there are other (better?) tools than dd for this, but sometimes I use dd exactly as you have. True image backups like this are required for data recovery situations where the filesystem is corrupt, or you are trying to recover deleted files. They are also required in the USA for "computer forensic analysis" used in courtroom presentations. Your assumption that dd will also get the content of /HDD2 is wrong. dd is a low level block copy tool. It has no understanding of the filesystem, and certainly not of mounts. The much bigger problem you have is that creating a dd copy of a running/operational filesystem is not a good way to backup anything. You have no way to know if the image you created is consistent or not. If you want true image backups, use your SUSE CDs in rescue mode (or the live CD). Boot from CD, then "mount /dev/some_other_drive /mnt", but don't mount /dev/hdb2 Then do your dd command, and you have a valid/solid image backup of your drive. Also, if you are doing this because your drive is unreliable, you need to properly handle disk read errors: dd if=/dev/hdb1 of=/mnt/image conv=noerror,sync noerror says to proceed copying even if an error is encountered. sync says that if a read error is incountered, write-out a block of zeros. The sync command ensures that image has exactly the same number of sectors as /dev/hdb1. If for some reason you doing this for forensic reasons, you then need to do a md5sum on both /dev/hdb1 and the image. The md5sum must agree for this to be considered a valid image. Greg -- Greg Freemyer
Thanks Greg Your first assumption applies, I want a disk image so I can plaster it back (somehow) onto the old, a replacement or another disk (on the same system) in the event the current drive goes corrupt on me. On XP machines I use Acronis True Image which also produces a boot disk for restoration purposes and will create the archive image file (a special compressed file taken from the disk image) which it uses for restoration. YES, I can use the SuSE 9.1 first disk as a boot up and go into the menus and then go into "rescue" mode. Kind of obvious now you mention it but it did not occur to me. Thanks very much for your suggestion. I would imagine in order to do a restore I simply use the same boot up mechanism but reverse the process again using the "dd" command with the "of=" and "if=" files reversed. However, you tweaked my curiosity on the two other considerations. 1. DISK READ ERRORS I understand the need to use zeros for an image creation in order to preserve the correct number of blocks (but it did not occur to be to do so). But this obviously leaves a data gap so how would one be aware of those gaps in the image and how would one remedy that situation so that one had a "complete -- without errors -- backup image or image plus some other kind of small backup (to cover the gap)??? Again, I never thought about having a disk read error during a backup but I know they can happen and so I guess in doing the backup I should make allowance for the possibility??? 2. For FORENSIC REASONS -- md5sum Could I have an example? I have never (to my knowledge) utilized md5sum, in point of fact I don't even know what it is and I will have to read up on it. Thanks again. Your input to my problem was very helpful especially "The much bigger problem you have..." and "Your assumption that dd will also get the content of /HDD2 is wrong". So you have addressed the entire set of issues as well as my false assumption and for that I am grateful. TED Greg Freemyer wrote:
On Wed, 29 Dec 2004 12:57:59 -0700, Ted Hilts
wrote: I used "dd if=/dev/hdb2 of=/dev/another_drive_on_same_machine/image_file" (where hdb2 contained the SuSE 9.1 installation with top directory root or "/" ) and from where I was executing the"dd" command (as root) in order to initially create a complete backup of the disk in the form of an image file located on "another_drive_on_same_machine" so that I could move it to another LAN computer capable of blasting it onto a DVD.
Now the "another_drive_on_same_machine" happens to be mounted as /HDD2 in the installation on /dev/hdb2. HDD2 is my drive for backups. It has to be mounted and therefore is also found in the directory path under "/".
This of course leads to the problem where "dd" (essentially a file copy utility) includes the content of /HDD2 and there is no "exclude" provision in the "dd" command. I could move the contents out of HDD2 (80 Gig drive) but I do not have an unused drive on the same machine and the drives on the other LAN machines are not able to take this additional load to use as a temporary storage.
Any ideas?
As others have said, there are other (better?) tools than dd for this, but sometimes I use dd exactly as you have.
True image backups like this are required for data recovery situations where the filesystem is corrupt, or you are trying to recover deleted files.
They are also required in the USA for "computer forensic analysis" used in courtroom presentations.
Your assumption that dd will also get the content of /HDD2 is wrong. dd is a low level block copy tool. It has no understanding of the filesystem, and certainly not of mounts.
The much bigger problem you have is that creating a dd copy of a running/operational filesystem is not a good way to backup anything. You have no way to know if the image you created is consistent or not.
If you want true image backups, use your SUSE CDs in rescue mode (or the live CD).
Boot from CD, then "mount /dev/some_other_drive /mnt", but don't mount /dev/hdb2
Then do your dd command, and you have a valid/solid image backup of your drive.
Also, if you are doing this because your drive is unreliable, you need to properly handle disk read errors:
dd if=/dev/hdb1 of=/mnt/image conv=noerror,sync
noerror says to proceed copying even if an error is encountered. sync says that if a read error is incountered, write-out a block of zeros. The sync command ensures that image has exactly the same number of sectors as /dev/hdb1.
If for some reason you doing this for forensic reasons, you then need to do a md5sum on both /dev/hdb1 and the image. The md5sum must agree for this to be considered a valid image.
Greg
On Fri, 31 Dec 2004 13:50:21 -0700, Ted Hilts
Thanks Greg
No problem, I do a lot of this kind of imaging. (My company does computer forensics.)
Your first assumption applies, I want a disk image so I can plaster it back (somehow) onto the old, a replacement or another disk (on the same system) in the event the current drive goes corrupt on me. On XP machines I use Acronis True Image which also produces a boot disk for restoration purposes and will create the archive image file (a special compressed file taken from the disk image) which it uses for restoration.
YES, I can use the SuSE 9.1 first disk as a boot up and go into the menus and then go into "rescue" mode. Kind of obvious now you mention it but it did not occur to me. Thanks very much for your suggestion. I would imagine in order to do a restore I simply use the same boot up mechanism but reverse the process again using the "dd" command with the "of=" and "if=" files reversed.
Correct.
However, you tweaked my curiosity on the two other considerations.
1. DISK READ ERRORS
I understand the need to use zeros for an image creation in order to preserve the correct number of blocks (but it did not occur to be to do so). But this obviously leaves a data gap so how would one be aware of those gaps in the image and how would one remedy that situation so that one had a "complete -- without errors -- backup image or image plus some other kind of small backup (to cover the gap)???
First if a read error occurs, the o/s will retry several times before it fails. Therefore, as far as I know, if dd itself is told of a read error by the kernel, there is nothing to be done. The data is already lost. The good news is that dd will tell you if it experiences an error and it will tell you what sector(s) had the problem, so you definately know what is going on. I don't know how to figure out what file contains that sector, but at least you have the rest of your data and as coherent of a filesystem as possible. FYI: The first thing data recovery companies do in the presence of a malfunctioning drive is to make a dd copy of the drive, then work with that image. FYI2: With modern drives, if you are getting bad blocks / read failures, it is time to replace the drive, but in our business we have to get data off of defective drives fairly often. This can mean swapping electronics, or opening up the drive in a clean room. I don't do any of that work, so I can't give any real details.
Again, I never thought about having a disk read error during a backup but I know they can happen and so I guess in doing the backup I should make allowance for the possibility???
2. For FORENSIC REASONS -- md5sum
Could I have an example? I have never (to my knowledge) utilized md5sum, in point of fact I don't even know what it is and I will have to read up on it.
For instance if I am making a forensic image, I would do: dd if=/dev/hdc conv=noerror,sync | split -a 3 -d -b 640m - backup-image. That will create a series of 640 MB files that comprise the backup. backup-image.000 backup-image.001 backup-image.002 backup-image.003 . . . FYI: We like to keep everything below the size of a CD. The reasons are historical and may no longer apply, but we still do it. I doubt if you would need to do this. Then we create an md5 checksum of the 640 MB segments: md5sum backup-image.* > md5 Then at various times during the analysis process we verify that all the 640MB segments are still valid and have not been corrupted. md5sum -c md5 If for some reason we need to restore the image to a drive: cat backup-image.* > /dev/hdc This works because all the 640 MB segments are in alphabetical order.
Thanks again. Your input to my problem was very helpful especially "The much bigger problem you have..." and "Your assumption that dd will also get the content of /HDD2 is wrong". So you have addressed the entire set of issues as well as my false assumption and for that I am grateful.
TED
Greg -- Greg Freemyer
participants (4)
-
Greg Freemyer
-
Jon Nelson
-
riccardo
-
Ted Hilts