Copy and Flatten directory contents with zsh

Question

I have a directory tree, where each subdirectory contains several different filetypes, I want to copy one particular filetype from each of the subdirectory, but I need to flatten the results, so that they all end up in one directory - only copying the newly added files and preserving permissions I have been using cp under zsh, with the following command line

cp -np **/*.ftype ../destination

Which has worked brilliantly up until now. However, I've just reached a limit, I'm not sure if it's the number of directories in the source directory (currently 217) or the total number of directories / desired files, (somewhere between 2672 and 2690) but suddenly I'm getting the error argument list too long: cp

I was hoping to use something like cp -np [A-Ma-m]**/*.ftype ../destination and split the job into parts, but I get no matches found: [A-Ma-m]**/*.ftype even though I know I have directories starting with the these ranges.

I've also tried

find Base_dir/ -iname '*.ftype' | xargs -J% cp -np % ../destination 

but it seems to break the directories at each point that there's a space character in the name, so it copies nothing.

I'm sure I'm doing something silly wrong, but any help would be appreciated.

Answer 1

The length of a command line in a shell is limited, probably to something like 65536 characters. So if your pattern expand to more characters than that, you'll get the "argument list too long" error you mentioned.

In such cases, find is the better approach (it will then also work for users of other shells). The copy can be accomplished with one of

find base_dir/ -iname '*.ftype' -print0 | xargs -0 -J% cp -np % dest_dir/
find base_dir/ -iname '*.ftype' -exec cp '{}' dest_dir/ \;

PS: The quoting in -iname '*.ftype' is important here to avoid expansion by the shell.

PPS: The error you get when using [A-Ma-m]**/*.ftype indicates that the pattern didn't get expanded. I don't know zsh very well, but most likely ** doesn't support additional patterns in front.

Answer 2

“Argument list too long” is due to the total length of the command line (the command name and the arguments, plus one byte for each — plus the environment, actually). It is limited to sysctl kern.argmax bytes (KERN_ARGMAX in the C sysctl interface). The usual way to get around this error is to split the command into multiple calls.

Zsh commes with a function called zargs which helps with automatically running a command multiple times with successive sets of arguments. It's similar to the external command xargs but a different syntax.

autoload -U zargs
zargs -i -- **/*.ftype -- cp -- {} ../destination

Put the autoload line in your ~/.zshrc to avoid having to type it each time you want to use it. (Some zsh configuration frameworks may do it for you.)

The option -i causes zargs to replace {} in the command by each argument in turn. This runs cp once per argument. zargs can also run the command with multiple arguments at a time, which is faster, but a limitation is that the grouped arguments have to go at the end of the command line. This is not a good match for cp since it needs the destination to go at the end.

GNU cp has an option -T which allows the destination to be passed before the sources, precisely to facilitate the use of xargs (and zargs which is similar). Assuming you have GNU cp installed as gcp:

zargs -- **/*.ftype -- cp -T ../destination --

Alternatively, an advantage of zargs over xargs is that you can use it on a function.

f () { $@[2,-1] $1 }
zargs -- **/*.ftype -- f ../destination cp --

Note how f only reorders its arguments, it doesn't have the cp -- … ../destination built in. The point is to keep the f call at least as long as the cp call, otherwise zargs would build a command line that goes over the limit when the cp -- and ../destination parts are added. Alternatively, pass a smaller limit:

f () { cp -- "$@" ../destination/; }
zargs -s $(($(sysctl kern.argmax) - $#functions[f])) -- **/*.ftype -- f

(or just type a limit, e.g. zargs -s 999999 -- **/*.ftype -- f).

Alternatively, stop worrying about the command line length limit and use the zcp function to do fancy copies. As before, put the autoload and alias lines in your .zshrc to avoid having to type them each time.

autoload -U zmv
alias zcp='zmv -C'
zcp '**/(*.ftype)' '../destination/$1'