RE: [SLE] Copying files
On Thursday 1 September 2005 at about 11:53 Darryl Gregorash wrote:
Albert wrote:
Midnight Commander? Umm, ok.. but then you have to sit there while all the work is being done :-) There may also be solutions which are even easier than the ones suggested so far.
files. Each image is 125x125 pixels named according to a numeric value indicating a number on a grid which in turn represents the world. These
Are they strictly numeric filenames?
Yes they are. Filenames similar to 3634719586.png. They start at 1.png and go to about 16987207681.png. In the directory we copied it was a subset of this.
Try something like
for a in {0..9}{0..9}{0..9}; do mkdir <target>/$a mv $a* <target>/$a/ done
This will create 1000 separate directories, eg. <target>/005/ etc, and move every file beginning with a specific combination of 3 numbers into a directory of that name, ie. 005* are all moved into <target>/005/. Each mv command will involve about 1000 files.
If you cannot handle moving about 1000 files at a time, then add another set of numbers: "for a in {0..9}{0..9}{0..9}{0..9}; ...." which would produce 10,000 directories of 100 files each.
If you do not want so many subdirectories in <target>, add another "for" layer. For example,
for a in {0..9}{0..9}; do mkdir <target>/$a for b in {0..9}{0..9}; do mv $a$b* <target>/$a done done
Now you get 100 directories with about 10,000 files in each, and move 100 files at a time. If that is too many files per directory, you can do all sorts of variants on this theme. This one will move 100 files at a time, and give you 100 directories containing 100 subdirectories, each containing approximately 100 files:
for a in {0..9}{0..9}; do mkdir <target>/$a for b in {0..9}{0..9}; do mkdir <target>/$a/$b for c in {0..9}{0..9}; do mv $a$b$c* <target>/$a/$b done done done
One of the above three will probably be the solution at the end of the day. It seems simple enough.
You may not be able to produce the exact file sorting you want with a command structure involving only "for" groups, but at least you will certainly be able to move them into a manageable tree structure, and then finish the work manually.
I am not too concerned about sorting the files in any specific order. Our software determines which image file should be used using a bit of maths. Once this is done, we know exactly which file is to be used. It can be easily altered to go look in subdirectories for the file. Thanks for the advice! Albert -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.344 / Virus Database: 267.10.18/86 - Release Date: 31/08/2005
On Thursday 01 September 2005 5:10 am, Albert wrote:
On Thursday 1 September 2005 at about 11:53 Darryl Gregorash wrote:
Albert wrote:
Midnight Commander? Umm, ok.. but then you have to sit there while all the work is being done :-) There may also be solutions which are even easier than the ones suggested so far.
files. Each image is 125x125 pixels named according to a numeric value indicating a number on a grid which in turn represents the world. These
Are they strictly numeric filenames?
Yes they are. Filenames similar to 3634719586.png. They start at 1.png and go to about 16987207681.png. In the directory we copied it was a subset of this. Would your company consider doing any testing of OSS software on this gargantuan number of files? Could be a way to give back by testing changes to ls, rm, find and others. You have mastered the testing environment quite well I'd say.
Run this idea by the Marketing department as a talking point for Sales. Maybe they could get you a quad, dual-core AMD x86_64 system with a SCSI RAID set to play with? Or a cluster of them with a Fibre Channel array? Get that processing time down to 36 minutes instead of 36 hours?
Albert
Just a thought, Stan
participants (2)
-
Albert
-
Stan Glasoe