On Mon, 24 Apr 2000, Gerhard Sittig wrote:
but it still is a lengthy process concerning time. What's The Right Way(TM) to go? I feel this kind of problem should have been around often enough so that there could be proven solutions.
Hello Gerhard,
today I got another reply from Valdis Kletnieks with a lot of more
detailed explanations. I forward the message.
Cheers, Peter
--
Peter Münster
http://gmv.spm.univ-rennes1.fr/~peter/
---------- Forwarded message ----------
Date: Mon, 24 Apr 2000 14:56:59 -0400
From: Valdis.Kletnieks@vt.edu
To: Peter Münster
+ find $TMP_DIR/. $OMIT ! -type d \ + -atime +$MAX_DAYS_IN_TMP -exec rm -f '{}' ';' + find $TMP_DIR/. $OMIT -depth -type d -empty -mindepth 1 \ + -mtime +$MAX_DAYS_IN_TMP -exec rmdir '{}' ';'
OK.. It must be summertime, and time for re-runs. This was thrashed to
death on Bugtraq some 6 years ago. I'm appending a re-run of the note
from LAST time... ;)
--
Valdis Kletnieks
Operating Systems Analyst
Virginia Tech
--- start appended note
Approved-By: Aleph One
From Redhat's /etc/crontab file: # Remove /var/tmp files not accessed in 10 days 43 02 * * * root find /var/tmp/* -atime +3 -exec rm -f {} \; 2> /dev/null
# Remove /tmp files not accessed in 10 days # I commented out this line because I tend to "store" stuff in /tmp # 41 02 * * * root find /tmp/* -atime +10 -exec rm -f {} \; 2> /dev/null
# Remove formatted man pages not accessed in 10 days 39 02 * * * root find /var/catman/cat?/* -atime +10 -exec rm -f {} \; 2> /dev/null
# Remove and TeX fonts not used in 10 days 35 02 * * * root find /var/lib/texmf/* -type f -atime +10 -exec rm -f {} \; 2> /dev/null
Folks, do NOT use 'find' on a public directory with '-exec rm -f' as root. Period. Ever. Delete it from your crontab *now* and finish reading the rest of this message later. * PROBLEM DISCUSSION AND EXPLOITATION The immediate security problem is that 'rm' doesn't check that components of the directory name are not symlinks. This means that you can delete any file on the system; indeed, with a little work you can delete *every* file on the system, provided that you can determine the file names (though you might be limited to deleting files more than ten days old). First, create the directories and file: /tmp/hacker-fest/some/arbitrary/set/of/path/names/etc/passwd where all but the last component is a directory. Be ready to replace 'etc' with a symlink to '/etc', so that: /tmp/hacker-fest/some/arbitrary/set/of/path/names/etc -> /etc i.e. the path components of the file name will point to a file named 'passwd' in a different directory. If the replacement operation occurs between when 'find' sets {} to "/tmp/hacker...etc/passwd" and when 'rm' calls unlink on "/tmp/hacker...etc/passwd", then rm will in fact delete '/etc/passwd', and not a file in /tmp. Deleting other files is left as an exercise. The race condition is really easy to win. Create a directory with 400 path components, like this: /tmp/hacker-fest/a/a/a/a/a/a/a.../a/a/a/etc/passwd (1) Then arrange for each of the 'a' components to be a symlink to a directory somewhere near the bottom of a similar tree. For example, /tmp/hacker-fest/a could be a symlink to /tmp/hacker-fest/b/b/b/b/b/b/b/b/b/.../b/b/b/b/b/b/a which could be a symlink to /tmp/hacker-fest/c/c/c/c/c/c/.../c/c/c/c/c/c/c and so on. In fact, *each* path component can be a symlink up to about 8 levels or so. Any operation such as stat(), open(), lstat(), etc. on one of these pathnames will cause the kernel to follow each and every symlink. The difference between lstat() and stat() in this case is that lstat() will not follow the *last* symlink. This will make lstat() and friends *extremely* slow, on the order of several *minutes* per lstat() operation, because each lstat() is now reading in several thousand inodes and disk blocks. If you fill each directory with several hundred entries, then create the entry you want, then delete the others, you force the kernel to waste its time reading kilobytes of empty directory blocks--in fact, you can make one stat() or unlink() operation read almost the entire disk in an order designed to maximize disk head motion if you know what you're doing. If you have an NFS, CDROM, or floppy-disk filesystem handy, you can get *weeks* per lstat(). Of course, 'find' will normally see the first symlink and stop. To prevent this, you rename the original directory (at (1) above) and create another directory with the same name and about 5000 empty files, some of which have the same name as files you want to delete. Note that these 5000 empty files can all be hard links to the same file, to save precious inodes for more of those symlinks. 'find' will spend considerable time iterating through these 5000 files. When it does (you'll be able to tell because the atime of the directory changes as find reads it), put the directory with the millions of symlinks at (1) back with a couple of rename operations. Some versions of 'find' will not be adversely impacted by this, but 'rm' definitely will. It is usually sufficient to simply create the 400-component-long directory, put 5000 files in it, wait for the atime of the directory to change, then do the rename so that 'rm' follows a symlink. I used this technique to remove /etc/crontab as a test case. If you have: /tmp/hacker-fest/a/a/a/a/a/.../a/etc/passwd (and 5000+ other files) /tmp/hacker-fest/a/a/a/a/a/.../a/usr where 'usr' is a symlink to '/usr', you can get some implementations of find to start recursing through /usr as well. * OTHER PROBLEMS WITH THIS CRONTAB A user can set the atime of any file they own to an arbitrary value, and that programs like zip, tar, and cpio will do this for you automatically; this makes 'atime' an almost useless indicator of when a file was last used ('mtime' has the same problem). Either the file will be deleted too early, because it was extracted from an archive using a program that preserves timestamps, or users can set the atime to well into the future and use /tmp space indefinitely. The later of ctime (to detect writes) and atime (to detect reads; must check that atime is not in the future) is a good indicator of when a file was last used. Miscellaneous bugs: the use of '*' means that files in a directory named '.foo' will never be cleaned (and you can prevent 'find' from working at all by putting more than 1020 files in /tmp). There are subdirectories of /var/catman that aren't properly handled by the 'find' command given (local and X11). You can't delete a directory with 'rm -f'. In other words, not only is RedHat's /etc/crontab a major security hole, it doesn't actually work properly, either. :( * FIXES The easiest way to fix this is to get rid of the find/rm stuff completely. If you need a garbage collector, try our LRU garbage collection daemon at the URL given below. Adding a system call that sets a flag that prevents a process from being able to ever follow a symlink would be non-portable, but efficient and effective. The next easiest way to fix this is to replace 'rm' with a program that does not follow symlinks. It must check that each filename component in turn by doing an lstat() of the directory, chdir() into the directory, and further lstat()s to check that the device/inode number of '.' is the same as the directory's device/inode number before chdir(). The parameter of the 'unlink' or 'rmdir' system call must not contain a slash; if it does, then the directory name before the slash can be replaced by a symlink to a different directory between verification of path components and the actual unlink() call. Another way to fix this is with a smarter version of find. A smart find does the chdir() and lstat() checks to make sure that it never crosses a symlink, and calls the program in 'exec' using a filename with no directory components, relative to the current directory. Thus, to delete: /tmp/hacker-fest/a/a/a/a/a/.../etc/passwd find first carefully (checking for attempts to exploit race conditions before and *after* each chdir()) chdir()s into /tmp/hacker-fest/a/a/a/a/a/.../etc and will fail if any of the components is a symlink, plugging the hole described above. After verifying that the '.../etc' is really a subdirectory of /tmp, and not some random point on the filesystem, find exec's the command: rm -f ./passwd which is secure as long as '.' isn't in your PATH. Note the leading './' to prevent rm from interpreting the filename as a parameter. Note: this is in *addition* to the checks that find already makes to determine whether a file is a symlink *before* chdir()ing into it. It must make sure that components of the path that have *already* been tested are not replaced with symlinks or renamed directories *after* find has started processing subdirectories of them. Note that the 'smart' find without the post-chdir symlink tests won't work. While smart-find is processing: /tmp/hacker-fest/a/a/a/a/* you can rename /tmp/hacker-fest/a/a/a/a to /tmp/hacker-fest/a/a/b (note: one less pathname component) and eventually smart-find will 'cd ..', but since the current directory of find has moved, '..' will move as well, and eventually smart-find will be one level too high and can start descending into other subdirectories of '/'. To help this along you may need to create: /tmp/hacker-fest/usr /tmp/hacker-fest/var etc. * SAFE LRU GARBAGE COLLECTION Our LRU /tmp garbage collector daemon is available at URL:http://www.ultratech.net/~zblaxell/admin_utils/filereaper.txt. It is implemented in perl5. It depends on a Linux-specific 'statfs()' system call to monitor available free space, so non-Linux people will need to do a port (send me patches and I'll incorporate them). Our garbage collector: handles the above security problems correctly, handles pathnames more than 1024 characters, uses smarter last-access estimates than just atime or ctime, can support "permanent" subdirectories, handles files, symlinks, directories, devices, mount points correctly, can support minimum age of files (e.g. no files < 1 day old), deletes oldest files first, deletes files only when disk space is low, and responds in less than ten seconds to low disk space conditions. Our garbage collector works on any directory where files can gracefully disappear at arbitrary times, such as /var/catman, /tmp, /var/tmp, TeX font directories, and our HTTP proxy cache. One directory where the garbage collector doesn't work very well is /var/spool/news; we had to hack things up a bit to fix the article databases when article files disappear. -- Zygo Blaxell. Former Unix/soft/hardware guru, U of Waterloo Computer Science Club. Current sysadmin for Myrus Design, Inc. 10th place, ACM Intl Collegiate Programming Contest Finals, 1994. Administer Linux nets for food, clothing, and anime. "I gave up $1000 to avoid working on windoze... *sigh*" - Amy Fong