2

I need to run a bash script on a list of subjects. I would like to build a while loop, that runs on multiple subjects at the same time. However, the commands within the loop must run sequentially.

May you please confirm me that a sintax like this would do the job?

file="subjects.txt"

foo () {

command $subj etc
command 2 $subj etc
command 3 $subj etc
}

while read subj; do foo "$subj" &
done <$file

Thank you in advance for your help.

Ramtin

Improve this question

1 Answer 1

Reset to default
1 
file="subjects.txt"

foo () {
  subj="$1"
  command $subj etc
  command 2 $subj etc
  command 3 $subj etc
}
export -f foo

parallel foo :::: "$file"

It will run n jobs in parallel (where n = number of CPU threads). To change that use parallel -j10 for 10 jobs in parallel.

Read chapter 1+2 of http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html (can be downloaded at https://doi.org/10.5281/zenodo.1146014). Your command line will love you for it.

Improve this answer
5
  • Thank you for your answer. Just to clarify - by job you mean the sequence of commands listed within the loop in that exact order for one iteration (subject), is that correct? Command 3 would not run before/in parallel with command 2 etc...
    – ram
    Commented May 2, 2020 at 9:23
  • This seems not to be sorting out the issue. The commands within the function are still running in random order, which of course results to error output, as some commands are depending on previous ones. Am I missing anything? I could not find anything in the book you recommended about sequential commands in functions.
    – ram
    Commented May 7, 2020 at 10:10
  • They are not run in random order. They are run sequentially and the are started in the order given in the input, but with multiple jobs run in parallel. So what you are seeing is something else. See if you can distill what you see down to an MCVE.
    – Ole Tange
    Commented May 7, 2020 at 11:06
  • Maybe you are confused because output is written as soon as the job stops and not in the order of the input. You can force the output to be in the order of the input by using --keep-order.
    – Ole Tange
    Commented May 7, 2020 at 11:09
  • Yes I apologise, my fault. One of the commands was not running because of another mistake, and to me appeared to be skept because no error message was showing up for that command. Thank you for your help.
    – ram
    Commented May 7, 2020 at 16:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .