On 10/31/18 8:23 AM, Per Jessen wrote:
Bernhard Voelker wrote:
On 10/30/18 5:36 PM, Per Jessen wrote:
This might be file system dependent, I'm not sure. I've been doing some tidying up and got stuck on a few directories with millions of files in them. 3+ million per directory. Doing a 'find' takes a very long time and also essentially chokes the system. I ended up writing a small utility using getdents() instead, much faster and the system remains operational.
I was just wondering if e.g. 'find' or 'ls' had some options that would limit the scope ? (not mtime etc).
Both find and ls use gnulib's FTS implementation instead of raw readdir and friends.
Is that relatively new? This is an elderly system, not up-to-date, running openSUSE 11. Another reason for cleaning it up.
Indeed, FTS support has been added in the 4.3 and newer, and as far as I remember it was installed aside the regular 'find' (alias 'oldfind' upstreams) as 'ftsfind'. Nowadays, 'oldfind' has gone and is only available in the upstream git development tree.
You didn't show us the command line you used, but I would guess that you used some options that require an additional stat(). E.g. "ls -l" needs to do an additional stat(). Likewise for coloring output etc. Furthermore, ls defaults to sort the output. Better use the -U option to stick to "directory order". The option -f includes -U.
Yes, Anton also suggested using '-f' yesterday, but my initial test did not show any improvement. (within 5-10min).
I think the fastest possible way would be '/usr/bin/ls -1U', thus ensuring that you don't use the 'ls' alias which is usually adding some options automagically.
For find, it is the same: did you only run 'find'/'find -print' or 'find -ls'? The former 2 should should be quite fast while the latter also requires a stat().
I almost certainly did 'find dir -mtime +365' - this would also require a stat(), but my code shows the stat() is not the problem.
Assuming you have quite regular file names, you could have find to create commands for you, e.g. moving into a per-day subdirectory $ find . -maxdepth 1 -mindepth 1 \ -type f -mtime +365 \ -printf 'mkdir -p "%TY/%Tm/%Td" && mv -n "%P" "%TY/%Tm/%Td/"\n' would produce commands like: mkdir -p "2015/12/25" && mv -n "file.1451011600" "2015/12/25/" mkdir -p "2015/12/25" && mv -n "file.1451008000" "2015/12/25/" mkdir -p "2015/12/25" && mv -n "file.1451004400" "2015/12/25/" mkdir -p "2015/12/25" && mv -n "file.1451000800" "2015/12/25/" mkdir -p "2015/12/24" && mv -n "file.1450997200" "2015/12/24/" mkdir -p "2015/12/24" && mv -n "file.1450993600" "2015/12/24/" which you could feed into a subsequent shell. (Obviously, it would be faster if you'd know and create the subdirectories upfront.) If you just want to delete them, the -delete option is your friend (also >v4.3). Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org