Copying files

Albert

31 Aug 2005 31 Aug '05

08:10

Good day, I am using SuSE 9.2 and SuSE 9.3. The partition in question is ReiserFS. We have an image set on disk that we wish to backup. Each image is about 20 Kb and I think there are more than a million files in the directory. I cannot get a directory listing – the shell falls over when running ls (I’m logged out after about 20 minutes of hard disk activity and nothing coming to the screen). I cannot remove the files – the same as running ls. I tried to put it all in a tarball – after 5 days I stopped it. The tarball was 2 GB at that time. I need to use Midnight Commander to copy the files – everything else falls over. After copying for 2 days, I had a power failure and stopped the process as the UPS was running out of power. At that point the machine was so slow it took 30 seconds to serve a page through Apache that normally took less than 1 second. Can anyone suggest a way of copying this image set to another hard disk or DVD that will not take forever to do? If possible with a progress bar – indicating how much files there are and how much files still need to be copied. I could easily regenerate the images (it took 8 days to do it) if there were a better way to save these files on disk to make it easier to manage. Any thoughts on this one? Albert -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.344 / Virus Database: 267.10.17/85 - Release Date: 30/08/2005

Show replies by date

Arun K. Khan

31 Aug 31 Aug

10:01

New subject: [SLE] Copying files

On Wed, 2005-08-31 at 10:10 +0200, Albert wrote:

...

Can anyone suggest a way of copying this image set to another hard disk or DVD that will not take forever to do? If possible with a progress bar – indicating how much files there are and how much files still need to be copied.

Albert

10:08

New subject: [SLE] Copying files

Thanks Arun. If I remember correctly this will mirror <current device> to <backup device>. Is there a way I can only copy a directory from <current device> to <backup device>? Albert -----Original Message----- From: Arun K. Khan [mailto:knura@yahoo.com] Sent: Wednesday, August 31, 2005 12:02 To: suse-linux-e@suse.com Subject: Re: [SLE] Copying files On Wed, 2005-08-31 at 10:10 +0200, Albert wrote:

...

Can anyone suggest a way of copying this image set to another hard disk or DVD that will not take forever to do? If possible with a progress bar – indicating how much files there are and how much files still need to be copied.

dd if=/dev/<current device> of=/dev/<backup device> make sure <backup device> is >= <current device> Look at dd_rescue (ddrescue RPM) if you encounter bad blocks on your HDD. -- Arun Khan (knura at yahoo dot com) Jone's Motto: Friends come and go, but enemies accumulate. -- Check the headers for your unsubscription address For additional commands send e-mail to suse-linux-e-help@suse.com Also check the archives at http://lists.suse.com Please read the FAQs: suse-linux-e-faq@suse.com -- No virus found in this incoming message. Checked by AVG Anti-Virus. Version: 7.0.344 / Virus Database: 267.10.17/85 - Release Date: 30/08/2005 -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.344 / Virus Database: 267.10.17/85 - Release Date: 30/08/2005

Ed Harrison

10:36

New subject: [SLE] Copying files

** Reply to message from Albert on Wed, 31 Aug 2005 12:08:59 +0200

...

Thanks Arun.

If I remember correctly this will mirror <current device> to <backup device>. Is there a way I can only copy a directory from <current device> to <backup device>?

Albert

Mount the target to /mnt. Go to desired directory, one level above the directory to backup. find ./home -xdev | cpio -pdv /mnt This will back up the entire /home directory to /mnt. Ed Harrison, Registered Linux User #199533 SuSE 9.3, Kernel 2.6.11 PolarBar Mailer 1.26

Arun K. Khan

11:59

New subject: [SLE] Copying files

On Wed, 2005-08-31 at 12:08 +0200, Albert wrote:

...

If I remember correctly this will mirror <current device> to <backup device>. Is there a way I can only copy a directory from <current device> to <backup device>?

Yes, dd will copy the whole device. To copy a specific directory, I would just "cp --archive" or "tar" it to a file on your new HDD as shown below: cd <to dir to copy> tar -c -S -f - . | (cd <to new dir> && tar -x -p -v -f -) HTH, -- Arun Khan (knura at yahoo dot com) The price of seeking to force our beliefs on others is that someday they might force their beliefs on us. -- Mario Cuomo

Carlos E. R.

14:13

New subject: [SLE] Copying files

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Wednesday 2005-08-31 at 17:29 +0530, Arun K. Khan wrote:

...

Yes, dd will copy the whole device. To copy a specific directory, I would just "cp --archive" or "tar" it to a file on your new HDD as shown below:

cd <to dir to copy> tar -c -S -f - . | (cd <to new dir> && tar -x -p -v -f -)

Remember that Alfred, the OP, already said that tar failed: |> I tried to put it all in a tarball - after 5 days I stopped it. The |> tarballwas 2 GB at that time. He is talking of more than a million files (10^6) in a single directory, each file about 20 Kb. - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFDFbsGtTMYHG2NR9URAs4AAKCFFIKz12HP2Jm88duhq8bbAdFnLgCbByJu QiC5sR3dDfMTMnOqHOEbcsM= =Muiz -----END PGP SIGNATURE-----

jdd sur free

15:37

New subject: [SLE] Copying files

Carlos E. R. wrote:

...

He is talking of more than a million files (10^6) in a single directory, each file about 20 Kb.

so if nor cp nor mv works (with regexp to act on a small subset), better redo the image files. but will he be able to _delete_ this number of files? just an idea, may be stupid. will "ln -s" works? that is i ever only one file of this group was really important, could it be possible to do a ln -s thisfile mylink and copy from mylink? jdd -- pour m'écrire, aller sur: http://www.dodin.net http://valerie.dodin.net http://arvamip.free.fr

Jerry Feldman

15:46

New subject: [SLE] Copying files

...

The Wednesday 2005-08-31 at 17:29 +0530, Arun K. Khan wrote:

...
Yes, dd will copy the whole device. To copy a specific directory, I would just "cp --archive" or "tar" it to a file on your new HDD as shown below:

cd <to dir to copy> tar -c -S -f - . | (cd <to new dir> && tar -x -p -v -f -)

Remember that Alfred, the OP, already said that tar failed: |> I tried to put it all in a tarball - after 5 days I stopped it. The |> tarballwas 2 GB at that time.

He is talking of more than a million files (10^6) in a single directory, each file about 20 Kb. I might possibly suggest trying to use rsync (but I'm afraid that it might also fall over). Another (although horrible) suggestion might be to use find. Assume that the source directory is mounted as /usr/localfoo and the target

On Wednesday 31 August 2005 10:13 am, Carlos E. R. wrote: directory is mounted on /mnt: cd /usr/local/foo find . -type f -exec cp -p {} /mnt \; This is assuming that /usr/local/foo does not have subdirectories. You can set appropriate flags in find to prevent recursion. Another possibility is to partition the output into a number of subdirectories: Assume that you may have directories: /mnt/a, /mnt/b, ... find . -name \[aA\] -exec cp -p {} /mnt/a \; In any case, as I mentioned, my solutions are not very good. Note that a 64-bit system might handle this better. -- Jerry Feldman Boston Linux and Unix user group http://www.blu.org PGP key id:C5061EA9 PGP Key fingerprint:053C 73EC 3AC1 5C44 3E14 9245 FB00 3ED5 C506 1EA9

Carlos E. R.

18:16

New subject: [SLE] Copying files

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Wednesday 2005-08-31 at 11:46 -0400, Jerry Feldman wrote:

...

...
Remember that Alfred, the OP, already said that tar failed: |> I tried to put it all in a tarball - after 5 days I stopped it. The |> tarballwas 2 GB at that time.

He is talking of more than a million files (10^6) in a single directory, each file about 20 Kb.

...

I might possibly suggest trying to use rsync (but I'm afraid that it might also fall over).

mc (midnight comander) doesn't fail: it runs as a turtle, but doesn't fail; so rsync "might" work.

...

Another (although horrible) suggestion might be to use find.

...

Assume that the source directory is mounted as /usr/localfoo and the target directory is mounted on /mnt: cd /usr/local/foo find . -type f -exec cp -p {} /mnt \; This is assuming that /usr/local/foo does not have subdirectories. You can set appropriate flags in find to prevent recursion. Another possibility is to partition the output into a number of subdirectories: Assume that you may have directories: /mnt/a, /mnt/b, ... find . -name \[aA\] -exec cp -p {} /mnt/a \;

The problem is that the moment the shell tries to expand the directory listing to give it to the command line, it fails, because the commnad line becomes huge and overflows the buffer. It is possible to work on a single file if the name is known before hand, delete, move, whatever. And perhaps on a bunch of them if the bunch is sizeable, I don't know. I'm tempted to try and create a big directory, but... :-?

...

In any case, as I mentioned, my solutions are not very good.

Note that a 64-bit system might handle this better.

I don't think so, it is a bash limit, not an architecture limit. - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFDFfQdtTMYHG2NR9URAjPPAJkBzAIZokgZl1IlfhBjyK5/lPvBWgCgjr9E 8g0jM2bgV78zSGafxlrH+ow= =zkjD -----END PGP SIGNATURE-----

Jerry Feldman

19:16

New subject: [SLE] Copying files

On Wednesday 31 August 2005 2:16 pm, Carlos E. R. wrote:

...

...
Assume that the source directory is mounted as /usr/localfoo and the target directory is mounted on /mnt: cd /usr/local/foo find . -type f -exec cp -p {} /mnt \; This is assuming that /usr/local/foo does not have subdirectories. You can set appropriate flags in find to prevent recursion. Another possibility is to partition the output into a number of subdirectories: Assume that you may have directories: /mnt/a, /mnt/b, ... find . -name \[aA\] -exec cp -p {} /mnt/a \;

The problem is that the moment the shell tries to expand the directory listing to give it to the command line, it fails, because the commnad line becomes huge and overflows the buffer.

It is possible to work on a single file if the name is known before hand, delete, move, whatever. And perhaps on a bunch of them if the bunch is sizeable, I don't know. I'm tempted to try and create a big directory, but... :-? In the case of find, it does work on a single file. Actually, my second example has an error: find . -name \[aA\]\* -exec cp -p {} /mnt/a \; Find performs its own expansions, the shell is not involved. In the "-exec cp -p {} /mnt/a \;" The {} refers to a single file. AFAIK, find uses the dirent structure to read the directory and recurse as necessary.

-- Jerry Feldman Boston Linux and Unix user group http://www.blu.org PGP key id:C5061EA9 PGP Key fingerprint:053C 73EC 3AC1 5C44 3E14 9245 FB00 3ED5 C506 1EA9

Donn Washburn

20:40

New subject: [SLE] Copying files

Jerry Feldman wrote:

...

On Wednesday 31 August 2005 2:16 pm, Carlos E. R. wrote: You can even try to a mounted file system partition no matter what filesystem. cp -Rp /directory1 /directory2 Just remember size matters. It will copy all files and subdirectories

man cp -p same as -- preserve=mode,ownership,timestamps -R, -r, --recursive - copy directories recursively

...

...
...
Assume that the source directory is mounted as /usr/localfoo and the target directory is mounted on /mnt: cd /usr/local/foo find . -type f -exec cp -p {} /mnt \; This is assuming that /usr/local/foo does not have subdirectories. You can set appropriate flags in find to prevent recursion. Another possibility is to partition the output into a number of subdirectories: Assume that you may have directories: /mnt/a, /mnt/b, ... find . -name \[aA\] -exec cp -p {} /mnt/a \;

The problem is that the moment the shell tries to expand the directory listing to give it to the command line, it fails, because the commnad line becomes huge and overflows the buffer.

It is possible to work on a single file if the name is known before hand, delete, move, whatever. And perhaps on a bunch of them if the bunch is sizeable, I don't know. I'm tempted to try and create a big directory, but... :-?

In the case of find, it does work on a single file. Actually, my second example has an error: find . -name \[aA\]\* -exec cp -p {} /mnt/a \; Find performs its own expansions, the shell is not involved. In the "-exec cp -p {} /mnt/a \;" The {} refers to a single file. AFAIK, find uses the dirent structure to read the directory and recurse as necessary.

-- 73 de Donn Washburn Hpage: " http://www.hal-pc.org/~n5xwb " Ham Callsign N5XWB Email : " n5xwb@hal-pc.org " 307 Savoy St. HAMs: " n5xwb@arrl.net " Sugar Land, TX 77478 BMW MOA #: 4146 - Ambassador LL# 1.281.242.3256 " http://counter.li.org " #279316

Jerry Feldman

20:51

New subject: [SLE] Copying files

On Wednesday 31 August 2005 4:40 pm, Donn Washburn wrote:

...

ou can even try to a mounted file system partition no matter what filesystem. cp -Rp /directory1 /directory2 Just remember size matters. It will copy all files and subdirectories

man cp -p same as -- preserve=mode,ownership,timestamps -R, -r, --recursive - copy directories recursively This is correct. I believe that the original poster, Albert, had a problem with cp. cp -Rp /directory1 /directory2 This will create the following directory tree: /directory2/Directory1/....

In any case using the form above also avoids problems with the shell and widlcard expansion. -- Jerry Feldman Boston Linux and Unix user group http://www.blu.org PGP key id:C5061EA9 PGP Key fingerprint:053C 73EC 3AC1 5C44 3E14 9245 FB00 3ED5 C506 1EA9

Carl William Spitzer IV

2 Sep 2 Sep

03:57

New subject: [SLE] Copying files

On Wed, 2005-08-31 at 15:40 -0500, Donn Washburn wrote:

...

Jerry Feldman wrote:

...
On Wednesday 31 August 2005 2:16 pm, Carlos E. R. wrote: You can even try to a mounted file system partition no matter what filesystem. cp -Rp /directory1 /directory2 Just remember size matters. It will copy all files and subdirectories

man cp -p same as -- preserve=mode,ownership,timestamps -R, -r, --recursive - copy directories recursively

Worked great. Thanks. -- ___ _ _ _ ____ _ _ _ | | | | [__ | | | |___ |_|_| ___] | \/

Jerry Feldman

31 Aug 31 Aug

19:38

New subject: [SLE] Copying files

On Wednesday 31 August 2005 2:16 pm, Carlos E. R. wrote:

...

...
In any case, as I mentioned, my solutions are not very good.

Note that a 64-bit system might handle this better.

I don't think so, it is a bash limit, not an architecture limit.

Yes and no. The ls problem that Albert had was not a bash problem, but was most likely an ls buffer issue. The 32-bit architecture limits has been improved in the 2.6 kernel, but significantly improved on 64-bit systems, but may not apply to ls or bash. The bash limit comes into play when expanding wild cards on the command line, so cp * <some directory> will fail, which is why I suggested using find which deals with files and wild cards much differently than does bash. -- Jerry Feldman Linux Expertise Center (PTAC-MA/TX) Hewlett-Packard Co. 550 King Street LKG2a-X2 Littleton, Ma. 01460 (978)506-5243

jdd sur free

20:09

New subject: [SLE] Copying files

Jerry Feldman wrote:

...

On Wednesday 31 August 2005 2:16 pm, Carlos E. R. wrote:

...
...
In any case, as I mentioned, my solutions are not very good.

Note that a 64-bit system might handle this better. I don't think so, it is a bash limit, not an architecture limit. Yes and no. The ls problem that Albert had was not a bash problem, but was most likely an ls buffer issue.

if so, a ls > list.txt could work? jdd -- pour m'écrire, aller sur: http://www.dodin.net http://valerie.dodin.net http://arvamip.free.fr

Jerry Feldman

20:27

New subject: [SLE] Copying files

...

Jerry Feldman wrote:

...
On Wednesday 31 August 2005 2:16 pm, Carlos E. R. wrote:

...
...
In any case, as I mentioned, my solutions are not very good.

Note that a 64-bit system might handle this better.

I don't think so, it is a bash limit, not an architecture limit.

Yes and no. The ls problem that Albert had was not a bash problem, but was most likely an ls buffer issue.

if so, a ls > list.txt could work? No, if it is an ls problem, it is the buffering in ls. The shell does not get involved with the internals of a command. The only difference between displaying the ls output to the screen or to a file is

On Wednesday 31 August 2005 4:09 pm, jdd sur free wrote: that when displaying to a console, you have the tty driver. The only bash limit that would come into play is when while cards are expanded on the command line. -- Jerry Feldman Boston Linux and Unix user group http://www.blu.org PGP key id:C5061EA9 PGP Key fingerprint:053C 73EC 3AC1 5C44 3E14 9245 FB00 3ED5 C506 1EA9

Jim Cunning

21:07

New subject: [SLE] Copying files

Today at 4:27pm, Jerry Feldman wrote:

...

...
Jerry Feldman wrote:

...
On Wednesday 31 August 2005 2:16 pm, Carlos E. R. wrote: [...] Yes and no. The ls problem that Albert had was not a bash problem, but was most likely an ls buffer issue.

if so, a ls > list.txt could work? No, if it is an ls problem, it is the buffering in ls. The shell does not get involved with the internals of a command. The only difference between displaying the ls output to the screen or to a file is

On Wednesday 31 August 2005 4:09 pm, jdd sur free wrote: that when displaying to a console, you have the tty driver. The only bash limit that would come into play is when while cards are expanded on the command line.

It would be interesting to see whether 'ls' fails on this huge directory when given the -U flag (don't sort).

Jerry Feldman

21:13

New subject: [SLE] Copying files

On Wednesday 31 August 2005 5:07 pm, Jim Cunning wrote:

...

It would be interesting to see whether 'ls' fails on this huge directory when given the -U flag (don't sort). Yeah, but it may also buffer. I have not looked at the sources for ls on Linux. -- Jerry Feldman Boston Linux and Unix user group http://www.blu.org PGP key id:C5061EA9 PGP Key fingerprint:053C 73EC 3AC1 5C44 3E14 9245 FB00 3ED5 C506 1EA9

jdd sur free

10:52

New subject: [SLE] Copying files

Albert wrote:

...

Can anyone suggest a way of copying this image set to another hard disk or DVD that will not take forever to do? If possible with a progress bar -- indicating how much files there are and how much files still need to be copied.

I beg you already tried cp-va <regexp> ?

...

I could easily regenerate the images (it took 8 days to do it) if there were a better way to save these files on disk to make it easier to manage.

why not, and sorting them in different folders :-) jdd -- pour m'écrire, aller sur: http://www.dodin.net http://valerie.dodin.net http://arvamip.free.fr

Carlos E. R.

12:37

New subject: [SLE] Copying files

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Wednesday 2005-08-31 at 10:10 +0200, Albert wrote:

...

I am using SuSE 9.2 and SuSE 9.3. The partition in question is ReiserFS.

We have an image set on disk that we wish to backup. Each image is about 20 Kb and I think there are more than a million files in the directory.

By "image" I assume you mean "graphic file", not a "partition image". I think Arun understood the second. While using dd for a backup method should work, you will have the same problems to read again the backup as you have with the original.

...

I cannot get a directory listing - the shell falls over when running ls (I'm logged out after about 20 minutes of hard disk activity and nothing coming to the screen).

It is interesting to note that while filesystems like reiserfs can handle a very large number of files in a single directory, tools like "ls" can't, they bump into a listing size limit somewhere. I think somebody said it was a command line limit of 32 Kb, I don't know. It will be in the list archive. I'd like to know if there are tools that can manage such directories easily.

...

I need to use Midnight Commander to copy the files - everything else falls over. After copying for 2 days, I had a power failure and stopped the process as the UPS was running out of power. At that point the machine was so slow it took 30 seconds to serve a page through Apache that normally took less than 1 second.

I would stick to mc, as it doesn't crash. However, I would try to subdivide the directory in several other directories; knowing if there is a pattern for the file names wouls help. For example, you could move all files starting with "A" to "directory/A", etc. That way size would be divided by 28. Or you can go one step further: AA, AB, AC, ...,. ZZ. This method is used, for example, by postfix to subdivide its queues in /var/spool/postfix. A script might do it. You can try first if "ls AA*" (or an appropiate pattern) works, or do it manually using mc pattern matching and tagging. Notice that if you _move_ files from the directory, handling the rest should become easier.

...

Can anyone suggest a way of copying this image set to another hard disk or DVD that will not take forever to do? If possible with a progress bar - indicating how much files there are and how much files still need to be copied.

I could easily regenerate the images (it took 8 days to do it) if there were a better way to save these files on disk to make it easier to manage.

Divide and conquer ;-) - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD4DBQFDFaSjtTMYHG2NR9URAshBAKCFR2AZEBAFuo8R9AzqV0BsbaGVKwCXbUd1 VI6LDEaP13YXmz0SQUWeSw== =yC2n -----END PGP SIGNATURE-----

6811

Age (days ago)

6813

Last active (days ago)

List overview

Download

19 comments

10 participants

participants (10)

Albert
Arun K. Khan
Carl William Spitzer IV
Carlos E. R.
Donn Washburn
Ed Harrison
jdd sur free
Jerry Feldman
Jerry Feldman
Jim Cunning

Copying files

Albert

Arun K. Khan

Albert

Ed Harrison

Arun K. Khan

Carlos E. R.

jdd sur free

Jerry Feldman

Carlos E. R.

Jerry Feldman

Donn Washburn

Jerry Feldman

Carl William Spitzer IV

Jerry Feldman

jdd sur free

Jerry Feldman

Jim Cunning

Jerry Feldman

jdd sur free

Carlos E. R.

tags

participants (10)