The problem
for f in $(find .)
combines two incompatible things.
find prints a list of file paths delimited by newline characters. While the split+glob operator that is invoked when you leave that $(find .) unquoted in that list context splits it on the characters of $IFS (by default includes newline, but also space and tab (and NUL in zsh)) and performs globbing on each resulting word (except in zsh) (and even brace expansion in ksh93 (even if the braceexpand option is off in older versions) or pdksh derivatives!).
Even if you make it:
IFS='
' # split on newline only
set -o noglob # disable glob (also disables brace expansion
# done upon other expansions in ksh)
for f in $(find .) # invoke split+glob
That's still wrong as the newline character is as valid as any in a file path. The output of find -print is simply not post-processable reliably (except by using some convoluted trick, as shown here).
That also means the shell needs to store the output of find fully, and then split+glob it (which implies storing that output a second time in memory) before starting to loop over the files.
Note that find . | xargs cmd has similar problems (there, blanks, newline, single quote, double quote and backslash (and with some xarg implementations bytes not forming part of valid characters) are a problem)
More correct alternatives
The only way to use a for loop on the output of find would be to use zsh that supports IFS=$'\0' and:
IFS=$'\0'
for f in $(find . -print0)
(replace -print0 with -exec printf '%s\0' {} + for find implementations that don't support the non-standard (but quite common nowadays) -print0).
Here, the correct and portable way is to use -exec:
find . -exec something with {} \;
Or if something can take more than one argument:
find . -exec something with {} +
If you do need that list of files to be handled by a shell:
find . -exec sh -c '
for file do
something < "$file"
done' find-sh {} +
(beware it may start more than one sh).
On some systems, you can use:
find . -print0 | xargs -r0 something with
though that has little advantage over the standard syntax and means something's stdin is either the pipe or /dev/null.
One reason you may want to use that could be to use the -P option of GNU xargs for parallel processing. The stdin issue can also be worked around with GNU xargs with the -a option with shells supporting process substitution:
xargs -r0n 20 -P 4 -a <(find . -print0) something
for instance, to run up to 4 concurrent invocations of something each taking 20 file arguments.
With zsh or bash, another way to loop over the output of find -print0 is with:
while IFS= read -rd '' file <&3; do
something "$file" 3<&-
done 3< <(find . -print0)
read -d '' reads NUL delimited records instead of newline delimited ones.
bash-4.4 and above can also store files returned by find -print0 in an array with:
readarray -td '' files < <(find . -print0)
The zsh equivalent (which has the advantage of preserving find's exit status):
files=(${(0)"$(find . -print0)"})
With zsh, you can translate most find expressions to a combination of recursive globbing with glob qualifiers. For instance, looping over find . -name '*.txt' -type f -mtime -1 would be:
for file (./**/*.txt(ND.m-1)) cmd $file
Or
for file (**/*.txt(ND.m-1)) cmd -- $file
(beware of the need of -- as with **/*, file paths are not starting with ./, so may start with - for instance).
ksh93 and bash eventually added support for **/ (though not more advances forms of recursive globbing), but still not the glob qualifiers which makes the use of ** very limited there. Also beware that bash prior to 4.3 follows symlinks when descending the directory tree.
Like for looping over $(find .), that also means storing the whole list of files in memory¹. That may be desirable though in some cases when you don't want your actions on the files to have an influence on the finding of files (like when you add more files that could end-up being found themselves).
Other reliability/security considerations
Race conditions
Now, if we're talking of reliability, we have to mention the race conditions between the time find/zsh finds a file and checks that it meets the criteria and the time it is being used (TOCTOU race).
Even when descending a directory tree, one has to make sure not to follow symlinks and to do that without TOCTOU race. find (GNU find at least) does that by opening the directories using openat() with the right O_NOFOLLOW flags (where supported) and keeping a file descriptor open for each directory, zsh/bash/ksh don't do that. So in the face of an attacker being able to replace a directory with a symlink at the right time, you could end up descending the wrong directory.
Even if find does descend the directory properly, with -exec cmd {} \; and even more so with -exec cmd {} +, once cmd is executed, for instance as cmd ./foo/bar or cmd ./foo/bar ./foo/bar/baz, by the time cmd makes use of ./foo/bar, the attributes of bar may no longer meet the criteria matched by find, but even worse, ./foo may have been replaced by a symlink to some other place (and the race window is made a lot bigger with -exec {} + where find waits to have enough files to call cmd).
Some find implementations have a (non-standard yet) -execdir predicate to alleviate the second problem.
With:
find . -execdir cmd -- {} \;
find chdir()s into the parent directory of the file before running cmd. Instead of calling cmd -- ./foo/bar, it calls cmd -- ./bar (cmd -- bar with some implementations, hence the --), so the problem with ./foo being changed to a symlink is avoided. That makes using commands like rm safer (it could still remove a different file, but not a file in a different directory), but not commands that may modify the files unless they've been designed to not follow symlinks.
-execdir cmd -- {} + sometimes also works but with several implementations including some versions of GNU find, it is equivalent to -execdir cmd -- {} \;.
-execdir also has the benefit of working around some of the problems associated with too deep directory trees.
In:
find . -exec cmd {} \;
the size of the path given to cmd will grow with the depth of the directory the file is in. If that size gets bigger than PATH_MAX (something like 4k on Linux), then any system call that cmd does on that path will fail with a ENAMETOOLONG error.
With -execdir, only the file name (possibly prefixed with ./) is passed to cmd. File names themselves on most file systems have a much lower limit (NAME_MAX) than PATH_MAX, so the ENAMETOOLONG error is less likely to be encountered.
Bytes vs characters
Also, often overlooked when considering security around find and more generally with handling file names in general is the fact that on most Unix-like systems, file names are sequences of bytes (any byte value but 0 in a file path, and on most systems (ASCII based ones, we'll ignore the rare EBCDIC based ones for now) 0x2f is the path delimiter).
It's up to the applications to decide if they want to consider those bytes as text. And they generally do, but generally the translation from bytes to characters is done based on the user's locale, based on the environment.
What that means is that a given file name may have different text representation depending on the locale. For instance, the byte sequence 63 f4 74 e9 2e 74 78 74 would be côté.txt for an application interpreting that file name in a locale where the character set is ISO-8859-1, and cєtщ.txt in a locale where the charset is IS0-8859-5 instead.
Worse. In a locale where the charset is UTF-8 (the norm nowadays), 63 f4 74 e9 2e 74 78 74 simply couldn't be mapped to characters!
find is one such application that considers file names as text for its -name/-path predicates (and more, like -iname or -regex with some implementations).
What that means is that for instance, with several find implementations (including GNU find on GNU systems²).
find . -name '*.txt'
would not find our 63 f4 74 e9 2e 74 78 74 file above when called in a UTF-8 locale as * (which matches 0 or more characters, not bytes) could not match those non-characters.
LC_ALL=C find... would work around the problem as the C locale implies one byte per character and (generally) guarantees that all byte values map to a character (albeit possibly undefined ones for some byte values).
Now when it comes to looping over those file names from a shell, that byte vs character can also become a problem. We typically see 4 main types of shells in that regard:
The ones that are still not multi-byte aware like dash. For them, a byte maps to a character. For instance, in UTF-8, côté is 4 characters, but 6 bytes. In a locale where UTF-8 is the charset, in
find . -name '????' -exec dash -c '
name=${1##*/}; echo "${#name}"' sh {} \;
find will successfully find the files whose name consists of 4 characters encoded in UTF-8, but dash would report lengths ranging between 4 and 24.
yash: the opposite. It only deals with characters. All the input it takes is internally translated to characters. It makes for the most consistent shell, but it also means it cannot cope with arbitrary byte sequences (those that don't translate to valid characters). Even in the C locale, it can't cope with byte values above 0x7f.
find . -exec yash -c 'echo "$1"' sh {} \;
in a UTF-8 locale will fail on our ISO-8859-1 côté.txt from earlier for instance.
Those like bash or zsh where the multi-byte support has been progressively added. Those will fall back to considering bytes that can't be mapped to characters as if they were characters. They still have a few bugs here and there especially with less common multi-byte charsets like GBK or BIG5-HKSCS (those being quite nasty as many of their multi-byte characters contain bytes in the 0-127 range (like the ASCII characters)).
Those like the sh of FreeBSD (11 at least) or mksh -o utf8-mode that support multi-bytes but only for UTF-8.
Interrupted output
Another problem with parsing the output of find or even find -print0 may arise if find is interrupted, for instance because it has triggered some limit or was killed for whatever reason.
Example:
$ (ulimit -t 1; find / -type f -print0 2> /dev/null) | xargs -r0 printf 'rm -rf "%s"\n' | tail -n 2
rm -rf "/usr/lib/x86_64-linux-gnu/guile/2.2/ccache/language/ecmascript/parse.go"
rm -rf "/usr/"
zsh: cpu limit exceeded (core dumped) ( ulimit -t 1; find / -type f -print0 2> /dev/null; ) |
zsh: done xargs -r0 printf 'rm -rf "%s"\n' | tail -n 2
Here, find was interrupted because it reached the CPU time limit. Since the output is buffered (as it goes to a pipe), find had output a number of blocks to stdout and the end of the last block it had written at the time it was killed happened to be in the middle of some /usr/lib/x86_64-linux-gnu/guile... file path, here unfortunately just after the /usr/.
xargs, just saw a non-delimited /usr/ record followed by EOF and passed that to printf. If the command had been rm -rf instead, it could have had severe consequences.
Notes
¹ For completeness, we could mention a hacky way in zsh to loop over files using recursive globbing without storing the whole list in memory:
process() {
something with $REPLY
false
}
: **/*(ND.m-1+process)
+cmd is a glob qualifier that calls cmd (typically a function) with the current file path in $REPLY. The function returns true or false to decide if the file should be selected (and may also modify $REPLY or return several files in a $reply array). Here we do the processing in that function and return false so the file is not selected.
² GNU find uses the system's fnmatch() libc function to do the pattern matching, so the behaviour there depends on how that function copes with non-text data.
