Calling shell functions with xargs

Question

I am trying to use xargs to call a more complex function in parallel.

#!/bin/bash
echo_var(){
    echo $1
    return 0
}
seq -f "n%04g" 1 100 |xargs -n 1 -P 10 -i echo_var {} 
exit 0

This returns the error

xargs: echo_var: No such file or directory

Any ideas on how I can use xargs to accomplish this, or any other solution(s) would be welcome.

Answer 1

Exporting the function should do it (untested):

export -f echo_var
seq -f "n%04g" 1 100 | xargs -n 1 -P 10 -I {} bash -c 'echo_var "$@"' _ {}

You can use the builtin printf instead of the external seq:

printf "n%04g\n" {1..100} | xargs -n 1 -P 10 -I {} bash -c 'echo_var "$@"' _ {}

Also, using return 0 and exit 0 like that masks any error value that might be produced by the command preceding it. Also, if there's no error, it's the default and thus somewhat redundant.

@phobic mentions that the Bash command could be simplified to

bash -c 'echo_var "{}"'

moving the {} directly inside it. But it's vulnerable to command injection as pointed out by @Sasha.

Here is an example why you should not use the embedded format:

$ echo '$(date)' | xargs -I {} bash -c 'echo_var "{}"'
Sun Aug 18 11:56:45 CDT 2019

Another example of why not:

echo '\"; date\"' | xargs -I {} bash -c 'echo_var "{}"'

This is what is output using the safe format:

$ echo '$(date)' | xargs -I {} bash -c 'echo_var "$@"' _ {}
$(date)

This is comparable to using parameterized SQL queries to avoid injection.

I'm using date in a command substitution or in escaped quotes here instead of the rm command used in Sasha's comment since it's non-destructive.

Answer 2

Something like this should work also:

function testing() { sleep $1 ; }
echo {1..10} | xargs -n 1 | xargs -I@ -P4 bash -c "$(declare -f testing) ; testing @ ; echo @ "

Answer 3

Seems I can't make comments :-(

I was wondering about the focus on

bash -c 'echo_var "$@"' _ {}
vs
bash -c 'echo_var "{}"'

The 1st substitutes the {} as an arg to bash while the 2nd as an arg to the function. The fact that example 1 doesn't expand the $(date) is simply a a side effect.

If you don't want the functions args expanded , use single single quotes rather than double. To avoid messy nesting , use double quote (expand args on the other one)

$ echo '$(date)' | xargs -0 -L1 -I {} bash -c 'printit "{}"'
Fri 11 Sep 17:02:24 BST 2020

$ echo '$(date)' | xargs -0 -L1 -I {} bash -c "printit '{}'"
$(date)

Answer 4

Maybe this is bad practice, but you if you are defining functions in a .bashrc or other script, you can wrap the file or at least the function definitions with a setting of allexport:

set -o allexport

function funcy_town {
  echo 'this is a function'
}
function func_rock {
  echo 'this is a function, but different'
}
function cyber_func {
  echo 'this function does important things'
}
function the_man_from_funcle {
  echo 'not gonna lie'
}
function funcle_wiggly {
  echo 'at this point I\'m doing it for the funny names'
}
function extreme_function {
  echo 'goodbye'
}

set +o allexport

Answer 5

@DennisWilliamson 's answer fails to use the -n 1 argument correctly and therefore fails when the input is a sequence on one line, e.g.:

DennisWilliamson's Method:

export -f echo_var 
echo 1 2 3 | xargs -n 1 -P 10 -I {} bash -c 'echo_var "$@"' _ {}

Output (Incorrect, ignores -n 1):

1 2 3

But I found a robust and simpler method which works with -n 1.

My Simpler/Robust Method (drops -I {} & trailing {}):

export -f echo_var;
echo 1 2 3 | xargs -n 1 -P 10 bash -c 'echo_var "$@"' _

Output (Correct):

1
2
3

Answer 6

You can create your own version of xargs that can handle functions.

usage example:

function my_echo() {
    echo "$@"
}
echo 1 2 3 | xargs my_echo
# xargs: my_echo: No such file or directory
echo 1 2 3 | xargs2 my_echo
# 1 2 3

xargs2 itself:

is_function() {
  if [[ $(type -t "${1?ensure_is_a_function: provide a command}") == "function" ]]; then
    return 0
  else
    return 1
  fi
}

function xargs2 {
  if is_function "$1"; then
    export -f "${1?}" # so subprocess bash can see it
  else
    echo "xargs2: use xargs" >&2
    return 1
  fi
  ARGS=$(printf "%q " "$@") # escape
  cat </dev/stdin | xargs bash -c "$ARGS \$@" _
}