I have a directory full of .gz, I want to expand each archive in parallel with GNU parallel. However I did not achieve anything.
I tried
parallel 'gunzip {}' ::: `ls *.gz`
parallel gunzip `ls *.gz`
with no results, bash tells me:
/bin/bash: archive1.gz: command not found
...
What am I doing wrong?
Thanks
I found this, which suggests using the --gnu flag:
parallel --gnu gunzip ::: *gz
If this works, you should either delete /etc/parallel/config or change its contents to --gnu rather than --tollef (as root):
echo "--gnu" > /etc/parallel/config
Also, never parse the output of ls., use globbing as I have above or find instead:
find . -name "*gz*" -print0 | parallel -q0 gunzip
ls is less of an issue when piping into GNU Parallel. As long as you do not have malicious users making filenames containing \n you are safe. In this case, however, ::: *.gz would probably be the best.
Commented
Jun 10, 2013 at 20:00
ls output changes between *nix flavors, and locale settings, it tends to break on any kind of white space, not just newlines, you willavoid a lot of pain if you get used to not parsing ls. Globbing is both simpler and safer.
for i in *; do command "$i"; done is easier to write and safer. Why would you want to learn an idiom that is harder and more risky? Try for i in $(ls) in a directory that contains files with spaces in their names. Seriously, parsing ls is almost always not the right or easiest way to do things. If globbing is not an option, use find. If you insist on using ls at least don't use a for loop, this is much safer: ls | while read i; do command "$i"; done at least that won't break on spaces.
Doing this works:
ls *.gz | parallel -t gunzip
The -t is optional but is useful as it shows you the commands that are executed on stderr.
I'm not sure you are doing anything wrong ::: should work (it's meant to be equivalent) but not even the examples in the man page work for me.
Update: the --gnu flag makes it work as terdon said.
